Category: Object storage

Posted on

33: GreyBeards talk HPC storage with Frederic Van Haren, founder HighFens & former Sr. Director of HPC at Nuance

IMG_6319In episode 33 we talk with Frederic Van Haren (@fvha), founder of HighFens, Inc. (@HighFens), a new HPC consultancy and former Senior Director of HPC at Nuance Communications. Howard and I got a chance to talk with Frederic at a recent HPE storage deep dive event, I met up with him again during SFD10, where he was talking on behalf of Kaminario, and he was also at HPE Discover conference last week.

Nuance is the backend speech recognition engine for a number of popular service offerings. Nuance looks very similar to a lot of other hyper-scale customers and ultimately, we feel may be the way of the future for all IT over the coming decades.  Nuance’s data storage journey since Frederic’s tenure with the company holds many lessons for all of us in the storage industry

Nuance currently has ~6PB usable (~16PB raw) of speech wave files as well as uncountable text and other files, all inside IBM SpectrumScale (GPFS).  They have both lots of big files and lots of small files. These days, Spectrum Scale is processing 2-3M files/second. They have doubled capacity for each of the last 9 years, and today handle a billion new files a month. GPFS stripes data across storage, provides data protection, migration, snapshotting and storage tiering across a diverse mix of storage. At the end of the podcast we discussed some open source alternatives to Spectrum Scale but at the time Nuance started down this path,  GPFS was found to be the only thing that could do the job. This proved to be a great solution as they have completely swapped out the underlying storage at least 3 times and all their users were none the wiser.

The first storage that Frederic talked about was Coraid (no longer in business) and their ATA over Ethernet storage solution. This used a SuperMicro with 24 SATA drives/shelf and they bought 40 shelves. Over time this grew to 1000s of SATA drives and was easily scaleable but hard to manage, as it was pretty dumb storage. In fact, they had to deploy video cameras, focused on drive shelves, to detect when drives failed!

Overtime, Nuance came to the realization that they had to do something more manageable and brought in HPE MSA storage to replace their Coraid storage. The MSA was a great solution for them which had 96 SAS drives, were able to support both faster “SCRATCH” storage using fast SAS 300GB/15KRPM drives and slower “STATIC” storage with slower SATA 760GB/7.2KRPM drives and was much more manageable than the Coraid solution.

Although MSA storage worked great, after a while, Nuance’s sprawling FC environment which was doubling yearly, caused them to rethink their storage once again. This led them to swap out all their HPE MSA storage, for HPE 3PAR to consolidate their FC network and storage footprint.

For metadata, Nuance uses a 76 node, Hadoop cluster for sophisticated search queries as doing an LS on the GPFS file system would take days. Their file meta-data is essentially a textual, row by row database and they use queries over the Hadoop cluster to determine things like which files have american english, spoken by females, with 8Khz recording.  Not sure when, but eventually Nuance deployed HPE Vertica SQL over Hadoop for their metadata engine and dropped average query from 12 minutes to 73 sec.(!!)

Nuance, because of their extreme growth and more open environment to storage innovation, had become a favorite for storage startups and major vendors to do Proofs of Concepts (PoC) on new storage offerings. One PoC, Nuance did was for Kamanario storage. There is a standard metric that says a CPU core requires so many IOPS, so that when CPU cores  increase,  you need to supply more IOPS. They went with Kaminario for their test-dev environment and more performance intensive storage. Nuance appreciates Kamanario’s reliability, high availability and highly predictable performance. (See the SFD10 video feed for Frederic’s session)

We talked a bit about how speech recognition’s Hidden Markov Chain statistical model was heavily dependent on CPU cores. Today, if you want to do a recognition task, you assigned it to one core and waited until it was done, a serial process dependent on the # of CPU cores you had available. This turned out to be quite a problem as you had to scale CPU cores if you wanted to do more concurrent speech recognition activities. Then came GPUs and you could do speech recognition work on a GPU core. With the new GPU cards,   instead of a server having ~16 CPU cores,  you could have a server with multiple Graphic cards having 3000-GPU cores. This scaled a lot easier. Machine learning and deep neural nets have the potential to parallelize this, so that it will scale even better

In the end, HPC trials, tribulations and ways of doing business are starting to become  mainstream. I was recently talking to one vendor that said, most HPC groups start out in isolation to support one application but over time they either subsume corporate IT or get absorbed into corp. IT or continue to be a standalone group (while waiting until one of the other two happen).

The podcast runs ~41 minutes and  covers a lot of ground about one HPC organization’s evolution of their storage environment over time, what was driving some of that evolution and the tools they chose to master it.  Listen to the podcast to learn more.

Podcast: Play in new window | Download (Duration: 40:35 — 18.6MB) | Embed

Subscribe: Apple Podcasts | Spotify | RSS

0F2A7849 - Copyv2-resizedFrederic Van Haren, founder HighFens, Inc.

Frederic Van Haren is the Chief Technology Officer @Highfens and known for his insights in the HPC and storage industry. He has over 20 years of experience in High Tech providing technical leadership and strategic direction in Telecom and Speech markets. Frederic spent the last decade at  Nuance Communications building large HPC environments from the ground up. He is frequently invited to speak at events to provide his insights on the HPC and storage markets. He has played leading roles as President of a variety of technology user groups promoting the use of innovative technology. As an Engineer he enjoys working with the engineering teams from technology vendors providing feedback on new and upcoming products.

Frederic lives in Massachusetts,  USA but grew up in the northern part of Belgium where he received his Masters in Electrical Engineering, Electronics and Automation.

Posted on

Greybeards talk car videos, storage and IT trends with Marc Farley

In our 30th episode, we talk with 3rd time guest star,  Marc Farley (@GoFarley), Formerly with Datera and Tegile. Marc has recently gone on sabbatical and we wanted to talk to him about what was keeping him busy and what was going on in storage/IT industry these days.

Marc is currently curating a car comedy vlog called theridecast.com. Apparently people, at least in California, are making comedy videos in their cars. They can be quite hilarious, checkout this episode of comedian in cars getting coffee.

While in the storage biz, the industry is getting battered by a number of trends: IT shrinking budgets, vendor proliferation, migration to cloud, and flash becoming old hat. Marc makes multiple points as to why the storage market is undergoing such a major transition these days:

  • Death to tech refresh, long live the cloud –  yes the cloud does upgrade hardware but  planned storage system obsolesce doesn’t happen in the cloud anymore. Cloud providers are  buying new SSDs, disks, white box servers, memory etc,  but not enterprise class storage, server or networking hardware.
  • AFA is boring, but selling – every vendor’s got one , two or sometimes three and they all know how to provide flash storage services. Customers pay extra for AFA, whether they need to or not, because they are swapping out old expensive, enterprise class storage for AFAs that often cost less but still provide better performance..
  • Tail IO latency becoming more important but it’s not understood – when IO response times go from 100µsec to 10msec, it hurts. It doesn’t matter if it’s every 1000 or 10,000 IOs, customers want less performance variability, which is a main reason they move to AFA in the first place. But not all AFA’s perform the same in tail latency and SSD controller/system architecture make a big difference.
  • Hybrid storage survives but only if you go big – hybrid storage economics makes sense only for large, diverse data repositories, that mix user directories, non-performance sensitive apps, and other structured and unstructured data in one data store.
  • Greenfield apps & secondary storage are moving to the cloud but migrating current apps to the cloud is difficult –  for new app development and archive storage, moving to or starting in the cloud is a no-brainer. Transitioning running enterprise class apps to the cloud is tough to do, that requires multiple skill sets and may never be successful. Hybrid  (cloud-on premises) enterprise class apps are too arduous to even contemplate.
  • Realtime analytics is emerging but data needs to be on flash – yes MapReduce is a batch activity which can uses lots of slow disk but there’s more to analytics than MR, and doing log analysis, in anything approaching realtime, one needs flash performance.
  • Optical’s persistence is great but who leaves data on the same technology for  20 years –  with magnetic and electronic storage densities going up every couple of years, who could afford keep data on the same optical technology that was 20 years old. Imagine using microfiche to keep PB of data today, inconceivable.

As for IT in general, one limiter of IT activity will become the lack of skilled engineers, specifically full-stack engineers and data scientists.

We ended our discussions on the economics of Samsung 3D NAND and Intel-Micron (IM) 3D Xpoint non-volatile memories. Both new semiconductor technologies are always long term investments. Today, Samsung is probably losing money on each 3D TLC NAND SSD it sells, but over time, as  fab yields improve, it should become cheap enough to make a profit. Similarly, 3D Xpoint may be costly to produce early on, but as IM perfect  their fab processes, the technology should become inexpensive enough to make oodles of $s for them. And there’s more technology changes to come.

The podcast runs just over 40 minutes and covers a lot of ground. Marc’s been in the IT almost as long as the GreyBeards and has a unique perspective on what’s happening today, having been with so many diverse, major and (minor) startup vendors throughout his tenure in the industry.  Listen to the podcast to learn more.

Podcast: Play in new window | Download (Duration: 40:45 — 9.3MB) | Embed

Subscribe: Apple Podcasts | Spotify | RSS

Marc Farley


Marc is a storage greybeard who has worked for many storage companies and is currently on sabbatical. He has written three books on storage including his most recent, Rethinking Enterprise Storage: A Hybrid Cloud Model and his previous books Building Storage Networks and Storage Networking Fundamentals.

In addition to his writing books he has been a blogger and podcaster about storage topics while working for EqualLogic, Dell, 3PAR, HP, StorSimple,  Microsoft, and others.

When he is not working, Marc likes to ride bicycles, listen to music, spend time with his family and dote on his cats. Of course there’s that car video curation…

Posted on

GreyBeards talk HPC storage with Molly Rector, CMO & EVP, DDN

oIn our 27th episode we talk with Molly Rector (@MollyRector), CMO & EVP of Product Management/Worldwide Marketing for DDN.  Howard and I have known Molly since her days at Spectra Logic. Molly is also on the BoD of SNIA and Active Archive Alliance (AAA), so she’s very active in the storage industry, on multiple dimensions and a very busy lady.

We (or maybe just I) didn’t know that DDN has a 20 year history in storage and in servicing high performance computing (HPC) customers. It turns out that more enterprise IT organizations are starting to take on workloads that look like HPC activity.

In HPC there are 1000s of compute cores that are crunching on PB of data. For Oil&Gas companies, it’s seismic and wellhead analysis; with bio-informatics it’s genomic/proteomic analysis; and with financial services, it’s economic modeling/backtesting trading strategies. For today’s enterprises such as retailers, it’s customer activity analytics; for manufacturers, it’s machine sensor/log analysis;  and for banks/financial institutions, it’s credit/financial viability assessments. Enterprise IT might not have 1000s of cores at their disposal just yet, but it’s not far off. Molly thinks one way to help enterprise IT is to provide a SuperComputer as a service (ScaaS?) offering, where top 10 supercomputers can be rented out by the hour, sort of like a supercomputing compute/data cloud.

We start early talking about DDN WOS: object store, which can handle archive to cloud or backend tape libraries. Later we discuss DDN ExaScaler and GridScaler, which are NAS appliances for Lustre and massively scale out, parallel file system storage, respectively.

Another key supercomputing storage requirement is  predictable performance. Aside from sophisticated QoS offerings across their products, DDN also offers the IME solution, a bump in the cable, caching system, that can optimize large and small file IO activity for backend DDN NAS scalers. DDN IME is stateless and can be removed from the data path while still allowing IT access  to all their data.

While we were discussing DDN storage interfaces, Molly mentioned they were working on an Omni Path Fabric.  Intel’s new Omni Path Fabric is intended to replace rack scale PCIe networks for HPC.

This months edition is not too technical and runs just over 45 minutes. We only got to SNIA and AAA at the tail end and just for a minute or two. Molly’s always fun to talk to, with enough technical smarts to keep Howard and I at bay, at least for awhile :). Listen to the podcast to learn more.

Podcast: Play in new window | Download (Duration: 45:14 — 20.7MB) | Embed

Subscribe: Apple Podcasts | Spotify | RSS

HeadshotMolly Rector, CMO and EVP Product Management & Worldwide Marketing,  DDN

With 15 years of experience working in the HPC, Media and Entertainment, and Enterprise IT industries running global marketing programs, Molly Rector serves as DDN’s Chief Marketing Officer (CMO) responsible for product management and worldwide marketing. Rector’s role includes providing customer and market input into the company’s product roadmap, raising the Corporate brand visibility outside traditional markets, expanding the partner ecosystem and driving the end-to-end customer experience from definition to delivery.

Rector is a founding member and currently serves as Chairman of the Board for the Active Archive Alliance. She is also the Storage Networking Industry Association’s (SNIA) Vice Chairman of the Board and the Analytics and Big Data committee Vice Chairman. Prior to joining DDN, Rector was responsible for product management and worldwide marketing as CMO at Spectra Logic. During her tenure at Spectra Logic, the company grew revenues consistently by double digits year-over-year, while also maintaining profitability. Rector holds certifications as CommVault Certified System Administrator; Veritas Certified Data Protection Administrator; and Oracle Certified Enterprise DBA: Backup and Recovery. She earned a Bachelor’s of Science degree in biology and chemistry.

Posted on

GreyBeards talk global storage with Ellen Rubin CEO & Laz Vekiarides CTO, ClearSky Data

In this edition we discuss ClearSky Data’s global storage service with Ellen Rubin (@ellen_rubin), CEO & Co-Founder and Laz Vekiarides (@lazvek), CTO & Co-founder of ClearSky Data.  Both Ellen and Laz have been around the IT industry for decades and Laz in particular was deeply involved in the development of EqualLogic storage systems both at Dell and at EqualLogic, prior to the acquisition.

ClearSky Data provides a global, primary storage service that connects edge device(s) in the data center that supply a read/write cache for iSCSI block storage to a point-of-presence (PoP) appliance in the metro area which uses cloud storage as its backend storage repository . We get into the technology later but essentially a customer pays a $/GB/month fee and all the edge, point-of-presence hardware and cloud storage repository is bundled into that monthly price.

The service is implemented as a two level caching service: level one at the edge is a cluster of 2U appliances with compute, DRAM and up to 24 SSDs and a dedicated metro-ethernet networking link to the PoP; level two at the PoP includes a dual HA server configuration with a JBOD with even more SSDs that has a direct link to Amazon Web Services Simple Storage Service (AWS S3).

Data is compressed, (inline or post-process) deduped and encrypted at the edge. Encryption keys are kept by the customer. Data written to the edge is synch-mirrored to the PoP and when the PoP fills up or the customer’s time interval has elapsed, their data is destaged to Amazon S3 which can then be replicated to other regions, where needed.

As part of their service, ClearSky Data also offers disaster recovery. As all customer data resides in S3, it can easily be supplied to another edge appliance (with the proper keys) at any other metro area location connected to one of their PoP’s.

ClearSky Data handles eventual consistency (not all copies of the data residing in  cloud storage may be the same) by versioning the cloud data objects and providing point-in-time consistency.

At the edge, the service can be deployed as a cluster of appliances that work together to support the IO workload and the PoP is configured to handle whatever IO workload is required in the metro area. Activity at the edge is heavy compute (compression, dedupe and encrypting all the data that comes in) and workload at the PoP is more IO bandwidth/networking based.

ClearSky Data currently has PoP’s in Las Vegas, Philadelphia and Boston with more on the way in the US. Today, ClearSky Data offer’s iSCSI interface protocols but have plans to provide FC, NFS and SMB support as well.  As we recorded the podcast, ClearSky Data’s  service was not quite GA yet, but were close.

Full Disclosure: Howard has worked for ClearSky in the past.

This months edition runs just under 41 minutes and gets into the business side and technical side of their service. Ellen provided the business view and Laz handled all the technical questions Howard and I threw at him. We hope you enjoy the podcast.

Podcast: Play in new window | Download (Duration: 40:42 — 37.3MB) | Embed

Subscribe: Apple Podcasts | Spotify | RSS

Ellen and Laz-orig copyEllen Rubin, CEO & Co-Founder ClearSky Data

Ellen Rubin is an experienced entrepreneur with a proven track record in leading strategy, market positioning and go-to-market for fast-growing companies. Most recently she was co-founder of CloudSwitch, a cloud enablement software company that was successfully acquired by Verizon in 2011. At Verizon, Ellen ran the cloud products group and was responsible for the strategy and roadmap for all cloud offerings.

Prior to founding CloudSwitch, Ellen was Vice President of Marketing at Netezza (NYSE: NZ), the pioneer and global leader in data warehouse appliances that power business intelligence and analytics at over 200 enterprises worldwide. As a member of the early management team at Netezza, Ellen helped grow the company to $130 million in revenues and a successful IPO in 2007. Ellen defined and created broad market acceptance of a new category, “data warehouse appliances,” and led market strategy, product marketing, complementary technology relationships and marketing communications.

Prior to Netezza, Ellen founded Manna, an Israeli and Boston-based developer of real-time personalization software. Ellen played a key role in raising over $18 million in venture financing from leading US and Israeli venture capital firms, recruiting the US-based management team and defining product and market strategy. Ellen began her career as a marketing strategy consultant at Booz, Allen & Hamilton, and holds an MBA from Harvard Business School and an undergraduate degree magna cum laude from Harvard College. She speaks regularly at industry events and has been recognized as one of the Top 10 Women in the Cloud by CloudNOW, as a Woman to Watch by Mass High Tech and Rising Star Entrepreneur by the New England Venture Capital Association.

Laz Vekiarides, CTO  & Co-Founder ClearSky Data

For over 20 years Laz Vekiarides has served in key technical and leadership roles delivering breakthrough technologies to market. Most recently, he served as the Executive Director of Software Engineering for Dell’s EqualLogic Storage Engineering group, where he led the development of numerous storage innovations and established the EqualLogic product line as a leader in host OS and hypervisor integration.

Laz joined Dell from EqualLogic, which was acquired in early 2008, where he was a member of the core leadership team – playing a key role in the company’s early success as a Senior Engineering Manager and Architect for the PS Series SAN arrays and host tools. Prior to EqualLogic, Laz held senior engineering and management positions at several companies including 3COM and Banyan Systems.

An occasional blogger, Laz frequently speaks at industry conferences, particularly in the areas of virtualization and data storage. He holds several storage technology patents, as well as a BSEE from Northeastern University, and an MSCS from the Worcester Polytechnic Institute.

Posted on

Greybeards talk about Storage Trends in our 2014 Yearend Podcast-part 3

In this our end of year video podcast Howard and I discuss some of the trends impacting the storage industry today.  Which include Cloud, SSD/Flash, software defined storage and converged server-storage systems and object storage.

The video comes in at a little more than 43 minutes and is available to be streamed, from Vimeo

or as a downloadable podcast in as a 3 videos.

Part 3 (~8 minutes) discusses the state of object storage and some of the trends impacting it’s adoption.

Podcast: Play in new window | Download (33.9MB) | Embed

Subscribe: Apple Podcasts | Spotify | RSS