Jim Handy, General Director at Objective Analysis and I were at Future of Memory and Storage 2026 Conference in Santa Clara earlier this month and there was a lot of news going around. Jim’s been on our show just about every year to discuss FMS news, And with the recent focus beyond flash, it’s even harder for one person to keep up.
New tech was the main topic discussed at the show but the one underlying thread was the AI supercycle and the switch from AI training to inference. The supercycle and inference seems to be turning the IT industry on it’s head.
Podcast: Play in new window | Download (Duration: 46:05 — 63.3MB) | Embed
Subscribe: Apple Podcasts | Spotify | RSS
HBM and DRAM were highlights of the show because they are in every accelerator/GPU. For whatever reason, there seems to be no price elasticity with respect to accelerators, DRAM or data centers for that matter.
So the worlds memory fabs are moving from relatively low margin DRAM to the higher margin HBM. Starving the rest of the IT world for memory. Jason mentioned that fab HBM capacity is all contracted for out to the end of 2027.
Speaking of which there was a lot of talk about KV (key-value) data offload (taking data out of HBM and storing it elsewhere while it’s unused) and the IO requirements needed to support an effective KV cache offload . One vendor said their current storage solution served 10M IOPs for a single HBF (high bandwidth flash storage) device but the real solution will need 100M 512B IOPs.
It turns out that token and KV cache elements are relatively small and using 4K blocks wastes a lot of space. And of course with 1000s of processor elements accessing the KV cache during token processing, thinking and other AI inference activities, if the KV data elements are going to be anywhere but in HBM they need to be accessed very quickly.
HBF, located on the accelerator board adjacent to HBM, was a hot topic last year but this year one could see some solutions coming to market. The one vendor discussed above talked about their liquid cooled HBF. But there were others with HBF storage of thier own.
On the other hand there were a number of storage vendors talking about taking KV cache out off the accelerator and onto storage on the PCIe bus and using NVMe SSDs. Not HBF but maybe another tier of KV caching. However there was lot of talk about CXL shared memory pooling that could be used as a shared KV cache to also do this. Unclear which tech (CXL memory pooling or NVMe SSDs) will win this race but it just might be a combination of both of them.
One vendor mentioned a new technology stacking DRAM directly ontop of (accelerator) compute modules. This would eliminate latency required to go off chip to an HBM and would use a chiplet interface. Doing so would create an intermediate tier of accelerator memory between compute SRAM and HBM, cheaper than SRAM and with more capacity.
Hyperscalers are sitting on a lot of DDR4 DRAM which they are salvaging and moving to shared memory pools behind CXL switches. Jason has been talking about this for years but it’s becoming a reality.
Jim mentioned that even without the AI supercycle driving data center build out there would likely have been a DRAM and NAND crunch coming anyway. This just happens every few years as fabs over produce driving prices down, which drives demand up and leads to fabs being unable to satisfy demand. Usually component price increases depress demand which allows the fabs enough time to increase production and start the cycle again.
It’s just that the AI supercycle has driven all this like it’s on steroids. And price increases have yet to dampen demand. The hyperscalers don’t seem to care the price they have to pay to power AI inference.
In fact some hyperscalers are going right to the fabs and contracting for all the chips coming off a fab line for a period of time. This seems to be the only way they can guarantee to support the compute needed for AI inference.
For once there wasn’t much talk about increasing capacity or 3D NAND layers at the show. In prior shows SSD capacity and 3D NAND layer counts were always news.
Last year we saw a glimmer of 3D layer increase backlash as the tradeoffs of more layers vs, more capacity per layer were discussed. This year 3D NAND layers and SSD capacities weren’t even a topic of discussion. The NAND and SSD world are moving to high bandwidth and high capacity flash, talking less about the technology behind them and their capacity.
Jim mentioned one reason for the reluctance to discuss 3D layers increasing was that 3D NAND suppliers were running out of space for stairways. Turns out each layer in 3D NAND (and most are over 100 layers today) has to have a separate electronic connection outside of it and this requires building a stairway with one stair per layer. And as you get to 100 to 200 or more layers, the stairways themselves start consuming too much NAND floor space, reducing capacity.
Finally, MRAM was another hot topic at the show. It turns out at current chip technology (2nm) levels, NOR gates out of normal TTL don’t work that well. MRAM seems impervious to this problem and is increasingly being used to replace TTL NOR gates in semiconductors.
MRAM has been a long time vendor at the show but mostly in niche solutions. For example, MRAM seems pretty impervious to cosmic rays so it’s increasingly used in RAD hardened applications such as in space, nuclear reactors, et al. But this is a major new use case that could broaden MRAM technology ecosystem beyond the niche players. Watch for more in this space.
Jim and the team at Objective Analysis have been busy this year churning out articles and in depth reports on NAND. memory, and compute business and recent technology innovations. Check out their website (link above) to learn more.
Jim Handy, General Director Objective Analysis
Jim Handy of Objective Analysis has over 35 years in the electronics industry, including 20 years as a leading semiconductor and SSD industry analyst. Early in his career he held marketing and design positions at leading semiconductor suppliers including Intel, National Semiconductor, and Infineon.
A frequent presenter at trade shows, Mr. Handy is known for his technical depth, accurate forecasts, widespread industry presence and volume of publication.
He has written hundreds of market reports, articles for trade journals, and white papers, and is frequently interviewed and quoted in the electronics trade press and other media.
He posts blogs at www.TheMemoryGuy.com, and www.TheSSDguy.com
Jim Handy of Objective Analysis has over 35 years in the electronics industry including 20 years as a leading semiconductor and SSD industry analyst. Early in his career he held marketing and design positions at leading semiconductor suppliers including Intel, National Semiconductor, and Infineon.