Ep. 034 - The Fight for Fast Tokens, TPU v7, Vera Rubin, and Engrams (AI Supply Chain, InferenceX)

Summary

SemiAnalysis uses this episode to connect model architecture, memory hierarchy, and chip economics into one inference-stack story. The first half digs into engrams as learned n-gram embeddings stored in a hash table so models can push some meaning-building earlier and offload more of the inference footprint to DRAM, SSD, or CPU-pinned host memory. The second half introduces AgentX, a system-level benchmark for real agentic coding traffic, and uses it to estimate profitability across different accelerators and serving stacks. They then compare TPU v7, arguing it is already highly cost-competitive and will benefit from broader external availability and a native torch backend. The Vera Rubin discussion centers on memory bandwidth, disaggregated serving, and lookup-table quantization, with the main takeaway that inference economics are increasingly determined by how well vendors can convert bandwidth and memory capacity into realized tokens per dollar.

Notes

• Engrams are framed as a learned n-gram embedding scheme trained into the model and retrieved by hash-table lookup at inference.
• The goal is to move some meaning construction earlier in the network and reduce wasted parameters.
• Engram modules can be overlapped with other compute because they only require token IDs as input.
• Placement is a trade-off: earlier layers help more, but later layers give more time to hide retrieval latency.
• Qwen Flash is cited as putting engrams in layer 2 so DRAM/SSD retrieval has time to arrive.
• V1/V4.1 Flash is described as using layer 1, with less room for overlap than Qwen’s setup.
• Offload uses UVA and moves engrams into CPU pinned host memory.
• HBM-resident engrams were tested, but the speaker says the result was not worth covering.
• The broader point is that HBM scarcity is pushing models toward offload to DRAM and SSD.
• DeepSeek’s MLA work is cited as another effort aimed at reducing KV-cache HBM usage.
• The architecture argument is that token IDs at the front of the model naturally fit overlap-friendly retrieval.
• LongCat from Mtoan is said to have independently converged on the same technique.
• HBM is described as the central bottleneck, with teams trying to offload as much as possible.
• The speaker contrasts US labs optimizing HBM bandwidth versus Chinese labs being forced by hardware limits.
• A China-specific memory stack is described around Sen 950 and a domestic HBM-like part called high Q.
• Chinese labs are said to be pushing smaller KVs, KV compression, and lower-tier memory offload.
• Engrams are presented as a natural extension of that memory-efficiency pressure, effectively giving layers for free via DRAM.
• The serving motivation is long-context and high-throughput workloads for coding agents.
• AgentX is introduced as a benchmark for agentic coding traffic rather than a chip-only test.
• The replay traces reportedly show theoretical cache hit rates above 98%, leaving the key question as realized hit rate in-system.
• At low concurrency, offload matters less; at 40–60 TPS and hundreds of sessions, it becomes important.
• The benchmark is built from captured SemiAnalysis employee coding sessions replayed against open-source inference servers.
• The workload shape is described as simple, tool-heavy, and cache-heavy.
• The stack described includes Nixl for KV transfer, VLM for CPU offload, DRAM-only offload, and the Dynamo router.
• The profit model combines traffic, utilization, and hardware TCO to estimate inference margins.
• One cited configuration says GB200 MBL72 can reach about 40.70% profit margin for Kim K3.
• Some DeepSig V4.1 flash configurations are said not to be profitable.
• At 60% utilization and a 30% Moonshot license fee, GB300 is said to approach a 50% margin.
• The P50 request is described as almost 100,000 input tokens.
• TPU v7 is described as Google’s first real purchasable accelerator through its own cloud.
• The result is that TPU v7 is the cheapest cost per million tokens for nearly the whole curve.
• The external TPU TCO is stated at about $0.21.
• TPU inference performance is presented as early-stage and likely to improve with software.
• The comparison is explicitly based on the open-source stack rather than Google’s internal serving stack.
• The speakers expect TPU capacity to be sold externally more broadly as V7 and V8 mature.
• They also expect a native torch backend for TPU to reduce software friction.
• Older TPUs are described as fixed-size systolic arrays that wasted compute through padding.
• The speakers reference roughly 3 million TPUs being produced as context for Google’s scale.
• InferenceX support is moving from a private fork to the public repo and persistent clusters.
• TPUs are reportedly being installed in neoclouds, enabling hosting outside GCP for the first time.
• Rubin is framed as incremental overall, but still meaningful at the interactivity points people actually serve.
• GB200/GB300 are described as bigger platform shifts than Rubin, with new CPUs, switches, and networking.
• Rubin may be easier to adopt because liquid cooling is already familiar.
• The speaker says Rubin’s biggest benefit comes from aggregate bandwidth, not just per-GPU capacity.
• Rubin includes a new hardware-accelerated path called LB/LD based on lookup-table quantization.
• The speakers say they do not yet see applications for the lookup-table path, but expect specialized lab uses.
• Rubin is expected to improve inference efficiency on a per-dollar, per-watt, or adjusted dollar-watt basis.
• The speakers say NVIDIA is not pricing Rubin aggressively because buyers will still purchase it.
• CPX is presented as a Rubin feature for disaggregated serving, splitting prefill and decode.
• Rubin’s main step up is said to be HBM3-to-HBM4 bandwidth, nearly a 3x jump.
• The curve is not uniform: the speaker cites about 3x at one point and about 1.4x at high throughput.
• Fast-token products are discussed as serving more than 100 tokens per second for premium workflows.