Ep. 017 - DeepSeek V4 and Huawei Ascend NPU Performance (InferenceX) | Kimbo Chen, Cam Quilici, Bryan Shan, Jordan Nanos

Summary

The conversation centers on how DeepSeek V4 changes the inference game relative to V3: a 1M context window, two sparse-attention variants, an embedding compressor, and sliding-window logic that together cut KV-cache usage dramatically. The guests explain that these model-side changes are only half the story; day-zero support depends on adapting kernels to shape changes like MHC dimensions and on having the right runtime access and vendor libraries in advance. They also unpack mega-kernel / mega-MOE ideas, where compute and communication are fused into a single kernel to reduce memory traffic and launch overhead, while noting the engineering tradeoffs and memory pressure this creates. The episode uses the InferenceX article to show how performance improves through many compounding optimizations rather than a single headline benchmark. It closes with broader ecosystem implications for vLLM, SGLang, Huawei Ascend, AMD FP4 enablement, and the competitive value of open-source tooling and documentation.

Notes

• DeepSeek V4’s headline upgrade is 1M context length, enabled by aggressive attention redesign rather than just scaling hardware.
• V4 introduces two sparse-attention variants: compressed sparse attention and heavily compressed attention.
• Those attention variants build on DeepSeek sparse attention from 3.2.
• Sparse query-key patterns reduce memory reads and make 1M context feasible without exploding memory requirements.
• An embedding compressor further shrinks each KV-cache entry.
• A sliding window is layered on top to reduce KV-cache usage even more.
• DeepSeek is said to be targeting roughly a 100x reduction in KV-cache usage versus a standard MOA model.
• Mega-MOE is framed as the other core V4 innovation, aimed at speeding expert FFN computation.
• Day-zero inference support required more than just knowing the architecture; hidden-size and dimension changes mattered too.
• A changed MHC dimension caused a day-zero hiccup for Nvidia.
• The implementation gap was shaped by access: vLLM and SGLang had NDA access, while Nvidia did not.
• Huawei’s optimization guidance was public, and the speakers describe it as a useful day-zero playbook.
• The team distinguishes mega-kernel from ordinary fusion: it breaks normal kernel boundaries.
• Mega-kernel aims to avoid register-to-HBM round trips by keeping work inside one kernel.
• The compute and communication overlap are fused into the same kernel in Mega-MOE.
• Aggressive fusion is not free: it requires substantial engineering effort.
• Fusion helps less when compute and communication already overlap at large batch sizes.
• Large fused kernels can raise memory pressure and even thermal risk by consuming more resources.
• The paper’s open-source Mega kernel was validated only on Nvidia GPUs and Huawei Ascend NPUs.
• The claimed speedup range for the open-source Mega kernel is 1.5x to 1.73x.
• On AMD, the biggest jump came from enabling FP4 support.
• AMD performance also improved by replacing Torch fallbacks with IREE or Triton kernels.
• The episode emphasizes compounding small optimizations, not just end-state benchmark numbers.
• InferenceX plans to add agentic benchmarks built from real internal code traces.
• Those future benchmarks are meant to surface features like Dynamo, KV Block Manager, and PD-disaggregation.
• Huawei’s stack is described as open source and hosted on GitCode.
• Documentation, meetups, and weekly calls are presented as part of Huawei’s developer engagement.
• The speakers say Huawei has sometimes implemented paper optimizations before Nvidia.
• DeepSeek and Z.ai/GLM serving on Ascend is taken as evidence that Huawei’s NPU ecosystem is getting stronger.
• The discussion suggests DeepSeek may be accepting very low or even zero margin to gain market share.
• The market context is China, where ByteDance is still viewed as the dominant AI/chat competitor.
• The release timeline was reportedly delayed multiple times, fueling speculation on X.
• The episode links the model design to real serving constraints like memory bandwidth and cache footprint.
• Kernel boundaries and launch overhead are treated as first-order performance bottlenecks.
• The performance story is framed as a systems problem spanning model design, kernel code, and vendor access.
• A practical takeaway is that support quality can hinge on who gets early access versus who must reverse engineer from scratch.
• Huawei’s public guidance suggests vendor ecosystems can be accelerated by transparent docs and community feedback loops.
• The crew treats open-source vs proprietary inference stacks as a strategic competition.
• The discussion implies that software maturity may matter as much as raw chip capability.
• DeepSeek V4’s design is presented as a bandwidth and memory optimization story as much as an accuracy story.
• The episode suggests that serving stacks win through stepwise improvements, not one magic kernel.
• The chat frames Ascend as a credible alternative in the China ecosystem, especially for serving.