Ep. 030 - Long Live the Short King: Why 4-HI HBM Wins (Memory) | Myron Xie, Jordan Nanos

Summary

Jordan Nanos and Myron Xie explain why the industry is revisiting the tradeoff between HBM bandwidth and capacity, and why 4-high stacks may win economically in a memory-constrained market. They say Nvidia’s Rubin Ultra roadmap appears to have moved from an early 1TB/package concept with four compute dies and 16-high HBM4E stacks to a 192GB HBM4 design with two compute dies and 8-high stacks. The core reason, they argue, is supply: DRAM wafer capacity is tight, HBM demand from accelerators is rising, and shipping less memory per package can maximize the number of accelerators delivered. They emphasize that once a system has enough memory for weights and KV cache, extra capacity has diminishing returns for inference, while bandwidth remains the real limiter. The episode also forecasts more custom SKUs, a potential shift of bottlenecks toward logic wafers and substrates, and a memory crunch that may not ease this decade.

Notes

• The episode’s central thesis is that 4-high HBM can outperform taller stacks on economics when bandwidth is the binding constraint.
• Myron frames the story as a shift from ever-taller HBM stacks toward a lower-capacity, higher-efficiency configuration.
• Rubin Ultra was originally previewed as a 1TB/package part.
• The revised understanding is far smaller: 192GB of HBM4, two compute dies, and 8-high stacks.
• The episode says the redesign is driven primarily by supply, not by a lack of performance headroom.
• The hosts argue that memory demand is rising faster than wafer supply can expand.
• Once memory exceeds the threshold needed for weights and KV cache, returns diminish.
• Jordan argues that the old assumption of ever-larger parameter counts is weaker now because performance can come from reasoning and other techniques.
• Myron says faster token generation is fundamentally a bandwidth problem.
• Post-training has become increasingly inference-like, changing what frontier labs need from hardware.
• Myron says HBM bandwidth is effectively fixed per stack, regardless of stack height.
• HBM pricing is described as capacity-based, so taller stacks cost more for the same bandwidth.
• Jordan points out that Rubin Ultra’s 192GB plan is below current 288GB configurations, which would be the first time a next-generation Nvidia flagship ships with less capacity than the part it replaces.
• Myron argues the supply constraint is shared across vendors, so others will likely follow Nvidia’s lower-memory approach.
• 4-high is presented as a manufacturability win because yield loss compounds less across fewer stacked dies.
• If 4-high materially expands cube supply, the bottleneck could move from memory to leading-edge logic wafers.
• Substrates and power delivery are also named as tight constraints.
• Conventional DRAM is also tight because wafers are being diverted toward HBM production.
• Myron says the memory supply response is slow because semiconductor capacity additions are long lead-time projects.
• The crunch is not expected to ease soon.
• Lower-memory variants are likely to proliferate as custom or semi-custom SKUs.
• The episode implies that different customers will receive different memory configurations based on workload needs and supply constraints.
• The broader strategic takeaway is that shipping more accelerators can matter more than maximizing memory per accelerator.
• The logic chain is: lower HBM per package, more packages shipped, and a better fit for bandwidth-heavy workloads.
• A 4-high stack is described as the practical floor for HBM; the industry cannot simply keep reducing stack height indefinitely.
• The hosts suggest the market is shifting from pretraining-dominated fleet growth to research, post-training, and inference.
• Myron ties the argument to parameter-scaling slowdowns and newer training/inference techniques.
• The capacity crunch is framed as a structural industry problem rather than a temporary cycle.
• The hosts reference a world where NVL72, NVL144, and NVL576 scale-up systems were part of the design imagination.
• The discussion implies that HBM stack-height choices are now workload optimization decisions, not just technology roadmaps.
• The episode repeatedly returns to a single operational goal: maximize shipped accelerators under scarce memory supply.
• The hosts say the memory shortage affects both HBM and conventional server DRAM.
• The supply problem is linked to equipment scarcity as much as fab space.
• The episode treats 4-high as a way to trade excess capacity for better shipped-unit economics.
• The hosts emphasize that the strategic question is not whether capacity matters, but when it stops mattering enough to justify more HBM.
• The practical conclusion is that frontier GPUs may ship with less memory than their predecessors for the first time in years.
• The episode suggests memory configuration is becoming a customer-specific product decision, not a one-size-fits-all spec.