Ep. 030 - Rubin Ultra Drops From 1 TB to 192GB of HBM (Memory)

Summary

The episode’s core thesis is that the memory market has shifted enough that taller HBM stacks no longer make the most sense for many frontier AI systems. Instead of maximizing bytes per package, vendors and customers are optimizing for total shipped accelerators, bandwidth per dollar, and scarce supply-chain resources. Nvidia’s Rubin Ultra is presented as the key case study, moving from an early terabyte-class concept to a much smaller 192 GB/package configuration built around 8-high HBM4. The speakers argue this is driven by DRAM scarcity, limited wafer additions, and the fact that inference and post-training now dominate demand. They also say model scaling trends and memory-skewed SKUs could fragment accelerator offerings further, with future systems right-sizing memory by workload rather than defaulting to the largest possible stack.

Notes

• The episode opens with the idea that AI memory design is shifting away from bigger-per-package HBM toward a lower-stack, system-optimized approach.
• Rubin Ultra was originally previewed as a 1 TB/package part.
• The original Rubin Ultra concept used four compute dies and 16 stacks of HBM4E.
• That original design assumed 32 Gbit, or 4 GB, per HBM4E die and 16-high stacks to reach the terabyte target.
• The newer Rubin Ultra understanding is far smaller: 192 GB of HBM4 per package.
• The package is now described as two compute dies instead of four.
• The launch configuration is said to use HBM4 rather than HBM4E.
• HBM4 is described here as 3 GB per die in an eight-high stack.
• The main driver is supply, not product preference.
• The speakers emphasize shortage across both DRAM and NAND, with DRAM as the key topic.
• Very little new DRAM wafer capacity is being added.
• Accelerator shipments are rising faster than HBM demand can be supplied.
• Customers and suppliers are rationing memory into more eight-high cubes instead of fewer twelve-high cubes.
• The argument is that limited HBM should be allocated across more packages to maximize shipped logic.
• For inference, HBM capacity matters only up to the point where weights and KV cache fit.
• Once that threshold is met, capacity gains have diminishing returns.
• The speakers separate capacity from bandwidth in token generation.
• Token speedups are framed as a bandwidth problem, not a capacity problem.
• Frontier compute is now dominated by inference and post-training, not classic pre-training.
• The marginal new data center buildouts are increasingly inference or research oriented.
• Rubin Ultra is also said to scale the system to NVL576.
• The larger system scale offsets the lower per-stack height by increasing aggregate HBM capacity in the domain.
• The engineering floor for full HBM bandwidth is described as four DRAM layers.
• HBM bandwidth is framed as fixed within each stack regardless of height.
• HBM4 and HBM4.0 are described as having 248 I/O data wires between cube and compute.
• Each cube can support up to 512 signal I/Os, so four cubes are needed to fully utilize the interface.
• HBM is priced on a dollar-per-gigabyte basis.
• The same bandwidth is available across stack heights, so 4-high has the best dollar-per-bandwidth economics.
• The episode says Rubin Ultra’s move to 192 GB is a reversal from the previous 288 GB-era shipping products.
• The speakers call this the first time their Nvidia tracking has seen a flagship GPU with less capacity than the prior generation.
• The capacity reduction is presented as a deliberate compromise to maximize the number of accelerators shipped.
• Roadmaps across vendors are said to have been reset for the new supply reality.
• Lower stack height is easier to manufacture because taller stacks compound yield loss.
• A simple example is given where 1% loss per layer compounds more badly across 8 or 12 layers than across 4.
• Four-die stacks are also easier for power delivery up the stack.
• The speakers conclude 4-high can produce more than double the harvestable cubes versus 8-high or 12-high.
• If HBM shifts to 4-high at scale, the bottleneck moves away from HBM wafers.
• The next constraints would likely be logic wafers, substrates, PCB, and power.
• TSMC logic wafer capacity is explicitly named as a potential co-packaging constraint.
• Substrates are identified as another tight supply-chain part.
• Power is also framed as a major downstream constraint.
• Relaxing HBM constraints would improve tokens per HBM wafer and help aggregate token output.
• The speakers analogize HBM wafers to other scarce resources like watts and dollars.
• Reducing HBM constraints is also framed as helping restore server DRAM availability.
• The semiconductor supply response is slow because of long lead times.
• Clean-room buildout is described as a primary reason short-term capacity cannot be added quickly.
• Tool availability is constrained by ASML supply chains and specialized component makers.
• The speaker’s view is that physical capacity constraints block a quick memory-supply fix.
• The memory crunch is not expected to ease within this decade.
• The counterargument is that very large future models could make higher-capacity HBM attractive again.
• If model sizes explode, extra memory capacity becomes more valuable for batching economics.
• Each user only needs one read of the weights in this batching framing, so larger weights magnify the value of extra capacity.
• Parameter growth is not increasing much because labs use techniques that preserve footprint.
• Loop transformers are cited as a newer technique to add compute depth without increasing model size.
• The labs are described as the loudest source of demand for high-memory accelerators.
• The speakers treat lab demand as a signal that model parameter sizes are not scaling as aggressively.
• Kimi K3 is invoked as a benchmark concern for OpenAI and Anthropic.
• Kimi K3 is said to be at 2.8 trillion parameters.
• The episode suggests multiple accelerator SKUs differentiated by memory may return.
• The historical pattern cited is V100 16 GB to 32 GB, A100 40 GB to 80 GB, and H100 80 GB to H200 144 GB.
• The speaker asks whether different customers could buy 4-high or 8-high versions of the same base logic die.
• The answer is yes: custom and semi-custom SKUs already exist.
• Meta is given as a concrete example of different memory from the mainstream configuration.
• MI450 is said to have a custom Meta version with 8-high HBM instead of 12-high.
• The rationale for SKU skewing is tighter supply-chain capacity and avoiding overprovisioning resources customers do not need.
• The episode places these SKU changes in the Rubin Ultra generation rather than in current Rubin.
• Rubin Ultra is characterized as the next meaningful hardware cycle after what is being deployed now.