WEKA's Val Bercovici: KV Cache, DeepSeek V4, HBF, SLC vs QLC NAND, CXL, NVLink, Tokenomics

Summary

Val Bercovici of WEKA and Vikram Sekar dig into how AI infrastructure is being rewritten by longer contexts, agent swarms, and exploding token usage. Val frames WEKA as an AI data and memory infrastructure company, arguing that inference is fundamentally memory-bound and that high-bandwidth fabrics can make remote memory paths competitive with or even faster than DRAM in some architectures. The conversation goes deep on KV cache compression, DeepSeek-style attention mechanisms, and why lower per-token memory use often increases total demand via longer contexts and more agents. They also debate NAND tiering, SLC versus TLC/QLC, high-bandwidth flash (HBF), and why Val thinks CXL has lost momentum relative to NVLink-style scale-up and RDMA-backed designs. The episode closes with a broader thesis: software is becoming compiled/run/inferenced, token OPEX is replacing traditional SaaS costs, and companies will increasingly need to own more of the token stack.

Notes

• AI memory and storage are being reshaped by longer contexts, agent swarms, and rising token demand.
• WEKA is positioned as AI data and memory infrastructure, not just storage.
• The discussion centers on inference-memory architecture rather than classical storage.
• Model routing to smaller models can reduce aggregate memory pressure.
• Routing smaller models is described as an easy button for lowering memory needs.
• Val says network-attached memory can beat DRAM in certain architectures.
• He argues the interconnect lane count can favor NVLink over CPU-to-DRAM paths.
• WEKA’s claim is that bandwidth can come from multiple fabrics, not a single vendor path.
• The team cites NVLink, Mellanox, BlueField, ConnectX, CPO, and other network options.
• Inference infrastructure is different from training infrastructure.
• Google’s TPU generations are cited as spanning both training and inference, but across different markets.
• WEKA’s augmented memory grid is presented as an inference-only product.
• It is described as backed by NAND at memory bandwidth.
• Val lays out a four-tier memory hierarchy.
• He emphasizes that the gaps between memory tiers are cliff-like.
• Without optimized compression, 100k tokens can require roughly 50 GB of KV cache.
• With optimizations, the same 100k tokens drop to about 5 GB of KV cache.
• He argues compression gains are overwhelmed by context and concurrency growth.
• Lower per-token memory use can simply be reinflated by larger contexts.
• The net result, he says, is more memory demand.
• Val expects context windows to keep growing beyond 1 million tokens.
• He says agent horizons are moving from hours to days and weeks.
• Agent swarms now dominate token volume.
• Chat sessions are becoming a smaller infrastructure concern.
• DeepSeek is used as the key example of cache-read pricing disruption.
• He says DeepSeek pricing is only available out of China.
• OpenRouter is cited as a way to compare hosted model pricing side by side.
• Logical cacheability can be high even when provider hit rate is not.
• He says provider-side cache hit rates depend on actual memory tiers available.
• DeepSeek-style compression is framed as combining multiple attention mechanisms.
• Sliding window attention compresses recent tokens more aggressively.
• He says DeepSeek V4 uses roughly five attention mechanisms.
• The result is about a 90% real-world reduction in KV cache per request.
• He argues the demand problem returns as requests get larger and faster.
• Inference is fundamentally more memory-centric than training.
• The goal is to make NAND behave more like memory than storage.
• Optane is called out as a missed fit for inference.
• SLC Flash is described as the endurance buffer when writes cannot be optimized enough.
• WEKA says it amortizes writes across a whole fabric of NVMe drives.
• That approach lets WEKA avoid needing SLC in many cases.
• Shared-everything storage is described as globally aware of queue depth via RAFT.
• The design goal is to shard and load-balance writes intelligently across many NVMe queues.
• Without SLC, overprovisioning can be severe.
• A petabyte may deliver only 300 to 500 TB of effective capacity in that scenario.
• HBF likely needs SLC in its common configuration.
• HBF is described as an offload tier that may replace DRAM and complement HBM.
• HBF packaging is the central product decision.
• Some vendors want HBF directly on the GPU package.
• Other vendors may put HBF on PCI cards or with ASICs instead of GPUs.
• KV transfer optimization and compiler expertise are presented as key to making HBF usable.
• Val predicts models may be recompiled to understand HBM and HBF tiers.
• Vic flags NAND endurance as a concern when flash is packaged next to GPUs or ASICs.
• Val frames endurance mitigation as a toolbox, not a single fix.
• The mitigation stack includes overprovisioning, NVMe-layer amortization, and explicit scheduling.
• Agent harnesses and eval loops can compress months of systems engineering into hours.
• But he says harness engineering is still needed for reliability and safety.
• Vic says some servers are only about half utilized on DRAM, motivating CXL interest.
• Val says he is no longer a CXL fan.
• He argues NVLink-style scale-up and Ethernet RDMA have reduced CXL’s need.
• Val says CXL would already be visible in major GPU roadmaps if it were a breakout technology.
• He cites a reported Vera Rubin BOM change as evidence DRAM is too expensive.
• Val says AMD MxT uses machine learning and small neural networks for caching.
• He says the current MxT use case is scientific computing.
• That work is being applied to life sciences, Monte Carlo, seismic, and weather workloads.
• Val says custom model training is much more accessible now.
• Domain-specific models may be small neural nets, not LLMs.
• He expects a Cambrian explosion of small-model applications.
• Robotics and edge inference are expected to stay mostly local and disconnected.
• He estimates 90% to 99% of robotics inference will be local.
• Cheap Raspberry Pi-style SoCs are making these use cases more practical.
• The AI flywheel is described as domain data plus model customization plus synthetic data loops.
• Val says the loop includes fine-tuning, distillation, quantization, and low-rank adaptation.
• He says software is shifting from compiled/run to compiled/run/inferenced.
• Token efficiency and model routing are becoming central to software economics.
• Val argues SaaS gross margins are being squeezed by token OPEX.
• He says the only way to stay profitable is to own more of the token stack.
• Val predicts SaaS companies and neo clouds will merge.