160: GreyBeard talks data security with Jonathan Halstuch, Co-Founder & CTO, RackTop Systems

Sponsored By:

This is the last in this year’s, GreyBeards-RackTop Systems podcast series and once again we are talking with Jonathan Halstuch (@JAHGT), Co-Founder and CTO, RackTop Systems. This time we discuss why traditional security practices can’t cut it alone, anymore. Listen to the podcast to learn more.

Turns out traditional security practices are keeping the bad guys out or supplies perimeter security with networking equivalents. But the problem is sometimes the bad guy is internal and at other times the bad guys pretend to be good guys with good credentials. Both of these aren’t something that networking or perimeter security can catch.

As a result, the enterprise needs both traditional security practices as well as something else. Something that operates inside the network, in a more centralized place, that can be used to detect bad behavior in real time.

Jonathan talked about a typical attack:

  • A phishing email link is clicked on ==> attacker now owns the laptop/desktop user’s credentials
  • Attacker scans the laptop/desktop for admin credentials or one time pass codes which can be just as good, in some cases ==> the attacker attempts to escalate privileges above the user and starts scanning customer data for anything worthwhile to steal, e.g. crypto wallets, passwords, client data, IP, etc.
  • Attacker copies data of interest and continues to scan for more data and to escalate privileges ==> by now if not later, your data is compromised, either it’s in the hands of others that may want to harm you or extract money from you or it’s been copied by a competitor, or worse a nation state.
  • At some point the attacker has scanned and copied any data of interest ==> at this point, depending on the attacker, they could install malware which can be easily detected to signal the IT organization it’s been compromised.

By the time security systems detect the malware, the attacker has been in your systems and all over your network for months, and it’s way too late to stop them from doing anything they want with your data.

In the past detection like this could have been 3rd party tools that scanned backups for malware or storage systems copying logs to be assessed, on a periodic basis.

The problem with such tools is that they always lag behind the time when the theft/corruption has occurred.

The need to detect in real time, at something like the storage system, is self-evident. The storage is the central point of access to data. If you could detect illegal or bad behavior there, and stop it before it could cause more harm that would be ideal.

In the past, storage system processors were extremely busy, just doing IO. But with today’s modern, multi-core, NUMA CPUs, this is no longer be the case.

Along with high performing IO, RackTop Systems supports user and admin behavioral analysis and activity assessors. These processes run continuously, monitoring user and admin IO and command activity, looking for known, bad or suspect behaviors.

When such behavior is detected, the storage system can prevent further access automatically, if so configured, or at a minimum, warn the security operations center (SOC) that suspicious behavior is happening and inform SOC of who is doing what. In this case, with a click of a link in the warning message, SOC admins can immediately stop the activity.

If it turns out the suspicious behavior was illegal, having the detection at the storage system can also provide SOC a list of files that have been accessed/changed/deleted by the user/admin. With these lists, SOC has a rapid assessment of what’s at risk or been lost.

Jonathan and I talked about RackTop Systems deployment options, which span physical appliances, SAN gateways to virtual appliances. Jonathan mentioned that RackTop Systems has a free trial offer using their virtual appliance that any costumer can download to try them out.

Jonathan Halstuch, Co-Founder & CTO, Racktop Systems

Jonathan Halstuch is the Chief Technology Officer and Co-Founder of RackTop Systems. He holds a bachelor’s degree in computer engineering from Georgia Tech as well as a master’s degree in engineering and technology management from George Washington University.

With over 20-years of experience as an engineer, technologist, and manager for the federal government, he provides organizations the most efficient and secure data management solutions to accelerate operations while reducing the burden on admins, users, and executives.

158: GreyBeards talk software defined storage with Brian Dean, Tech. Mkt., Dell PowerFlex

Sponsored By:

This is the 2nd time Brian Dean, Technical Marketing, Dell PowerFlex Storage has been on our show discussing their storage. Since last time there’s been a new release with significant functional enhancements to file services, Dell CloudIQ integration and other services. We discussed these and other topics on our talk with Brian. Please listen to the podcast to learn more.

We began the discussion on the recent (version 4.5) changes to Powerflex for file services. PowerFlex file services are provided by File Nodes each running a NAS Container, which supplies multiple NAS Servers. NAS servers supply tenant network namespaces, security policies and host file systems, each of which resides on a single PowerFlex volume.

File Nodes are deployed in HA pairs, each on a separate hardware server. One can have up to 16 File Nodes or 8 pairs of File Nodes running on a PowerFlex cluster. If one of the pair goes down, file access fails over to the other File Node in a pair.

Each NAS Server supports multiple file systems each of which can be up to 256TB. The NAS Container is also used for other Dell storage file services, so it’s full featured and very resilient.

PowerFlex file services support multiple NFS and SMB versions as well as SFTP/FTP and other essential file data services. In addition, it also supports a global name space which allows all PowerFlex cluster file systems to be accessed under a single name space and IP target.

Next, we discussed PowerFlex’s automated LCM (Life Cycle Management) services which is specific to the PowerFlex appliance and fully-integrated, rack deployment models. Recall that PowerFlex can be deployed as an appliance, rack solution or in a software only solution using X86 servers.

With the appliance and rack models, a PowerFlex Manager (PFxM) service is used to deploy, change, monitor and manage PowerFlex cluster nodes. It discovers networking and PowerFlex servers/storage, loads appropriate firmware, BIOS, PowerFlex storage data services software and then brings up PowerFlex block services.

PFxM also offers automated LCM by maintaining an intelligent catalog, which declares all current software/firmware/BIOS and hardware versions compatible with PowerFlex software. When changes are made to the cluster, say when storage is increased or a server is added, the PFxM service detects the change and goes about bringing any new hardware up to proper software levels.

Finally the PFxM service can non-disruptively update the cluster whenever a PowerFlex code change is deployed. This would involve an intelligent catalog update, after which the PFxM service detects the cluster is out of compliance, and then it would serially go through, bringing each cluster node up to the proper level, without host IO access interruption.

Finally, we discussed changes made to CloudIQ-PowerFlex interface, so that CloudIQ can now troubleshoot and report performance-capacity trends at the PowerFlex storage pool, fault set, and fault domain level. Previously, CloudIQ could only do this at the full PowerFlex system level.

CloudIQ is Dell’s free, cloud service used to monitor and trouble shoot all Dell storage systems and many other Dell solutions, whether on premises or in the cloud.

Brian mentioned that all technical information for PowerFlex is available on their InfoHub.

Brian Dean, Dell PowerFlex Technical Marketing

Brian is a 16+ year veteran of the technology industry, and before that spent a decade in higher education. Brian has worked at EMC and Dell for 7 years, first as Solutions Architect and then as TME, focusing primarily on PowerFlex and software-defined storage ecosystems.

Prior to joining EMC, Brian was on the consumer/buyer side of large storage systems, directing operations for two Internet-based digital video surveillance startups.

When he’s not wrestling with computer systems, he might be found hiking and climbing in the mountains of North Carolina. 

152: GreyBeards talk agent-less data security with Jonathan Halstuch, Co-Founder & CTO, RackTop Systems

Sponsored By:

Once again we return to our ongoing series with RackTop Systems, and their Co-Founder & CTO, Jonathan Halstuch (@JAHGT). This time we discuss how agent-less, storage based, security works and how it can help secure many organizations with (IoT) end points they may not control or can’t deploy agents on them. But agent-less security can also help other organizations with security agents deployed over their end points. Listen to the podcast to learn more.

The challenge for enterprise’s with agent based security, is that not all end points support them. Jonathan mentioned one health care customer with an older electron microscope that couldn’t be modified. These older, outdated systems are often targeted by cyber criminals because they are seldom updated.

But even the newest IoT devices often can’t be modified by organizations that use them. Agent-less, storage based security can be a final line of defense to any environment with IoT devices deployed.

But security exposures go beyond IoT devices. Agents can sometimes take manual effort to deploy and update. And as such, sometimes they are left un-deployed or improperly configured.

The advantage of a storage based, agent-less security approach is that it’s always on/always present, because it’s in the middle of the data path and is updated by the storage company, where possible. Yes, not every organization may allows this and for those organizations, storage agent updates will be also require manual effort.

Jonathan mentioned the term Data Firewall. I (a networking novice, at best) have always felt firewalls were a configuration nightmare.

But as we’ve discussed previously in our series, RackTop has a “learning” and an “active” mode. During learning, the system automatically configures application/user IO assessors to characterize normal IO activity. Once learning has completed, the RackTop Systems in the environment now understands what sorts of IO to expect from users/applications and can then flag anything outside normal IO patterns.

But even during “learning” mode, the system is actively monitoring for known malware signatures and other previously characterized bad actor IO. These assesors are always active. 

Keith mentioned that most organizations run special jobs on occasion (quarterly, yearly) which might have not been characterized during learning. Jonathan said these will be flagged and may be halted (depending on RackTop’s configuration). But authorized parties can easily approve that applications IO activity, using a web link provided in the storage security alert.

Once alerted, authorized personnel can allow that IO activity for a specific time period (say Dec-Jan), or just for a one time event. When the time period expires, that sort of IO will be flagged again.

Some sophisticated customers have change control and may know, ahead of time, that end of quarter or end of year processing is coming up. If so, they can easily configure RackTop Systems, ahead of time, to authorize the applications IO activity. In this case there wouldn’t be any interruption to the application.

With RackTop Systems, security agents are centrally located, in the data path and are always operating. This has no dependency on your backend storage such as, SAN, cloud, hybrid storage, etc., or any end point. If anything in your environment accesses data, those RackTop System assessors will be active, checking IO activity and securing your data. 

Jonathan Halstuch, Co-Founder and CTO, RackTop Systems

onathan Halstuch is the Chief Technology Officer and co-founder of RackTop Systems. He holds a bachelor’s degree in computer engineering from Georgia Tech as well as a master’s degree in engineering and technology management from George Washington University.

With over 20-years of experience as an engineer, technologist, and manager for the federal government he provides organizations the most efficient and secure data management solutions to accelerate operations while reducing the burden on admins, users, and executives.

147: GreyBeards talk ransomware protection with Jonathan Halstuch, Co-Founder and CTO, RackTop Systems

Sponsored By:

This is another in our series of sponsored podcasts with Jonathan Halstuch (@JAHGT), Co-Founder and CTO of RackTop Systems. You can hear more in Episode 145.

We asked Jonathan what was wrong with ransomware protection today. Jonathan started by mentioning that bad actors had been present, on average, 277 days in an environment before being detected. That much dwell time, means they could have easily corrupted most backups and snapshots, stolen copies of all your most of sensitive/proprietary data, and of course, encrypted all your storage.

Backup ransomware protection works ok if dwell time is a couple of days or even a week, but not multiple months or longer.. The only real solution to this level of ransomware sophistication is real time monitoring of IO, looking for illegal activity. Listen to the podcast to learn more

Often, any data corruption, when discovered, is just notification to an unsuspecting IT organization that they have been compromised and lost control over their systems. Sort of like having a thief ring the door bell to tell you they stole all your stuff after the fact.

The only real solution to data breaches and ransomware attacks with significant dwell time, that protects both your data and your reputation is something like RackTop Systems and their BrickStore SP storage system. BrickStore offers an ongoing, in real-time, active defense against ransomware that’s embedded in your data storage, that’s continuously looking for bad actors and their activities during IO activity, all day, every day. 

When BrickStor detects ransomware in progress it shuts it down, by halting any further access to that user/apllication and snapshots the data before corruption, to immutable snapshots. That way admins have a good copy of data.

In addition, RackTop BrickStor SP supplies run book like recovery procedures that tell IT how to retrieve good data from snapshots, without wasting valuable time searching for the “last good backup”, which could be months old.

I asked whether data at rest encryption could offer any help. Jonathan said data encryption can thwart only some types of attacks. But it’s not that useful for ransomware, as bad actors who infiltrate your system masquerade as valid users/admins and by doing so, gain access to decrypted data.  

RackTop Systems uses AI in its labs to create ransomware “assesors”, automated routines embedded in their storage data path, which continuously execute looking for bad actor IO patterns. It’s these assessors that provide the first line of defense against ransomware.

In addition to assessors, Racktop Systems supplies many reports which depict data access permissions, user/admin access permissions, data being accessed, etc. All of which help IT and security teams better understand how data is being used and provide the visibility needed to help support better cyber security

When ransomware is detected, RackTop BrickStor offers a number of different notification features that range from web-hooks and slack channels to email notices and just about everything in between to notify IT and security teams that a breach is occurring and where.

RackTop Systems BrickStor SP is available in many deployments. One new option, from HPE, uses their block storage to present LUNs to BrickStor SP. Jonathan mentioned that other enterprise class block storage vendors are starting to use BrickStor SP to supply secure NAS services for their customers as well.

Jonathan mentioned that RackTop attended the HIMSS conference in Chicago last week and will be attending many others throughout the year. So check them out at a conference near you if you get a chance.

Jonathan Halstuch, Co-Founder & CTO RackTop Systems

Jonathan Halstuch is the Chief Technology Officer and co-founder of RackTop Systems. He holds a bachelor’s degree in computer engineering from Georgia Tech as well as a master’s degree in engineering and technology management from George Washington University.

With over 20-years of experience as an engineer, technologist, and manager for the federal government he provides organizations the most efficient and secure data management solutions to accelerate operations while reducing the burden on admins, users, and executives.

116: GreyBeards talk VCF on VxBlock 1000 with Martin Hayes, DMTS, Dell Technologies

Sponsored By:

This past week, we had a great talk with Martin Hayes (@hayes_martinf), Distinguished Member Technical Staff at Dell Technologies about running VMware Cloud Foundation (VCF) on VxBlock 1000 converged infrastructure (CI). It used to be that Cloud Foundation required VMware vSAN primary storage but that changed a few years ago. . When that happened, the Dell Technologies team saw it as a great opportunity to support VCF on VxBlock CI.

This is the first GreyBeards podcast for Martin, but he was extremely knowledgeable about VxBlock and Cloud Foundation technologies. He’s been a technical product manager on the VxBlock converged infrastructure at Dell Technologies for many years. He’s an expert on Cloud Foundation and he knows an awful lot more about VMware NSX-T networking than seems reasonable (good thing). In any case, Martin’s expertise covers the whole gamut of VCF services as well as VxBlock 1000 infrastructure. The podcast is a bit longer than our normal sponsored podcast but there was a lot of information to cover. Listen to the podcast to learn more.

With VCF enabling primary storage on networked storage systems, all the storage vendors in the world gave a mighty cheer. But VMware Cloud Foundation still requires the vSAN servers to run its management domain. Late in 2020, VxBlock 1000 from Dell Technologies released a new software defined version of its Advanced Management Platform (AMP) to run on vSAN Ready Nodes. AMP is VxBlock’s management platform but also runs management domains for VCF and NSX-T.

For workload domains, VxBlock 1000 offers Cisco UCS M5 rack and blade servers, that can be configured to support just about any workload needed by a data center.

Historically, VMware vSphere problems with DR weren’t as much storage replication issues as networking problems. But NSX-T and VCF seemed to have solved that problem.

And with vRealize Automation plugins and NSX-T APIs, customers can have 0 touch network provisioning which enables the use of IaaS or infrastructure as code for their data center.

VMware vVOLs are now available with Dell EMC PowerMax storage. So, now VxBlock 1000 customers can use vSphere storage policy-based management (SPBM) as well as automated vVOL replication for data on PowerMax.

VMware NSX-T implements Application Virtual Networks (AVNs) using a GENEVE overlay network, which make extensive use of encapsulation. But where there’s encapsulation, de-encapsulation must follow to access outside networks. All this (encapsulation on ingress, de-encapsulation on egress) is done through NSX-T Edge clusters.

The net result of all this is that VMware customers have more choice, i.e., now they can run VCF on HCI or CI. And with VxBlock 1000 CI, VCF customers can select a best of breed components for each level of their 3-tier infrastructure.

Martin Hayes, DMTS, Dell Technologies

Martin Hayes is a Technical Product Manager at Dell Technologies, where he develops and executes data center product strategies that incorporate virtualization, software-defined networking (SDN) and converged systems.

Previously, he served in network advisory and architect roles at Dell EMC, converged systems pioneer VCE and Irish broadband provider eircom.