Ep. 034 - Engrams: How DeepSeek Offloads KV Cache to DRAM and SSD (Core Research) | Jordan Nanos, Cam Quilici, Alec Ibarra, Bryan Shan

Summary

Jordan Nanos, Cam Quilici, Bryan Shan, and Alec Ibarra center the episode on inference-system design: Engrams, DRAM/SSD offload, and the KV-cache bottlenecks that arise in agentic coding workloads. They describe Engrams as learned n-gram-style token embeddings that can be stored in a hash table and retrieved at inference time, with the core motivation being to reduce HBM pressure and move more of the memory burden to cheaper tiers like DRAM. The team then walks through Agent X, a replay benchmark built from real internal agentic coding traces, to measure cache reuse, prefill/decode behavior, and offloading under realistic traffic. They also discuss profitability for open-source inference on current high-end GPUs, arguing that under their TCO and pricing assumptions GB300-class serving can be highly lucrative. Later segments compare TPU v7 against Nvidia hardware, frame Vera Rubin as an incremental but important bandwidth-driven upgrade, and conclude that ultra-fast token generation only matters when it shortens the full tool-using workflow rather than just the model’s raw decode time.

Notes

• Episode framing: the discussion spans Engrams, offloading, Agent X, TPUs, Vera Rubin, fast tokens, and chip startups.
• Engrams are presented as learned 2-gram and 3-gram token embeddings.
• The key design is inference-time lookup from a hash table rather than recomputing everything in deeper layers.
• DeepSeek-style co-design moves meaning construction earlier in the network.
• The team says Engram layers can be asynchronously overlapped because they only need token IDs as input.
• Placement earlier in the network helps utility but can reduce overlap opportunities.
• DeepSeek and related work are framed as responses to limited HBM.
• HBM scarcity is the core bottleneck behind KV compression and offload.
• Engram data is moved to CPU pinned host memory using UVA for offload.
• Agent X replaces chip-level microbenchmarks with replayed agentic traffic.
• The benchmark models how tool calls append outputs back into the conversation stream.
• Million-context agent sessions create rapidly growing KV caches.
• Replay traces reportedly show theoretical cache hit rates above 98%.
• The practical question is how much of the theoretical cache hit rate the system can realize.
• DRAM offload becomes important in high-throughput scenarios like offline batch inference and RL rollouts.
• Their offload argument is that DRAM can deliver 20-30% cache hit rate instead of HBM in some workloads.
• The profit estimator combines accelerator TCO, OpenRouter pricing, and concurrency/throughput assumptions.
• One example cited about GB300MBR72 serving Kimik 3 yields roughly 40.70% margin.
• Cam argues the open-source inference stack can be a money printer at 60% utilization even after license fees.
• Jordan says the real-world traffic plus actual hardware cost model makes the latest open-source inference stack look very profitable.
• Cam goes further and claims a GB300 rack can pay back in a couple of months.
• TPU v7 is described as Google’s first real purchasable accelerator through its own cloud.
• They benchmark TPU v7 directly against B200 and B300.
• Alec says TPU v7 is the cheapest cost per million tokens on most of the Pareto curve.
• The TPU TCO assumption cited is about $0.21 externally.
• Cam expects TPU performance to improve quickly as the software stack matures.
• The guests expect TPU externalization to broaden access beyond internal Google workloads.
• The current TPU software stack still lacks a native Torch backend.
• TPUs are framed as systolic-array accelerators optimized for data movement.
• TPU externalization matters because it creates a third serious competitor to AMD and Nvidia.
• Jordan expects Inference X support to move from a private fork into the public repo.
• TPUs are also being installed in NEO clouds, expanding hosting beyond GCP.
• Vera Rubin is treated as an incremental hardware step that still matters for serving efficiency.
• Jordan says Rubin is more incremental than Blackwell-to-GB200 transitions, but still meaningful.
• Rubin shifts emphasis toward higher memory bandwidth and more aggregate memory across more GPUs.
• Rubin introduces a hardware-accelerated lookup-table quantization path referred to as LUDD/LB.
• The quantization path uses 3-bit keys and 8-bit lookup values, around 3.125 bits per weight.
• Jordan says Rubin is roughly 3x better around the middle of the inference curve but less dramatic at very high throughput.
• Rubin adds CPX for decode/prefill splitting, but the team does not yet have CPX support in Inference X.
• Jordan says Rubin’s main win is the HBM3-to-HBM4 jump in memory bandwidth.
• He argues Rubin is still better on a per-dollar or per-watt basis for inference.
• Bryan personally says he does not use fast mode much because he runs multiple agents in parallel.
• Cam argues tool execution dominates latency, so faster token generation often does not change end-to-end workflow time.
• Alec agrees that fast mode helps for quick responses, but not for most parallelized work.
• Cam says ultra-fast output may not justify its price if tool latency remains the bottleneck.
• Jordan raises the counterfactual of models becoming much bigger and slower, making speed more critical.
• Bryan says scaling laws still matter, but the industry is also innovating architecturally.
• Cam predicts fast tokens will be specialized and routed by task.
• Bryan says efficiency gains tend to get consumed by more usage, so smaller-model futures may not materialize.
• Cam notes that even parallel-agent users would still notice if responses slowed dramatically.
• Jensen Huang’s line is invoked: faster tokens are smarter tokens.
• Jordan says Tyler RT and similar optimizations may allow 200-500 tokens per second even on large models.
• Cam defines Tyler RT as a latency-optimized inference engine.
• Bryan describes Tyler RT as one giant mega-kernel and frames low-interactivity inference as a dataflow problem.
• Bryan says AI agents are well-suited to kernel development because the task is easy to verify.
• Jordan says chip startups should focus on demand, deployment, yields, servers, and data-center execution.
• He argues high interactivity can itself produce high throughput.
• Jordan says batch-one can still be efficient if the hardware is fast enough and pipelined.
• Cam is skeptical of chip startups that still depend on Nvidia GPUs for prefill.
• Jordan favors disaggregated prefill/decode pools that can be rebalanced as workloads change.
• Jordan explicitly argues for separate prefill and decode silicon.
• He says the RTL and software moat is real but less prohibitive than before.
• Cam envisions Inference X as a free real-world inference comparison hub.