Ep. 027 - OpenAI Jalapeño: Better Than Nvidia Blackwell (Accelerators)

Summary

The hosts discuss SemiAnalysis’ Jalapeño article and argue that OpenAI’s custom accelerator has already posted stronger perf-per-megawatt and perf-per-TCO results than Nvidia’s Blackwell/GB300 and Vera Rubin in the July benchmark window they chose for comparison. They stress that OpenAI cares about internal tokens per watt and total system economics, not external per-package chip metrics, because power—not chip count—is increasingly the binding constraint in data centers. A major technical theme is that Jalapeño appears to get more realized HBM bandwidth out of HBM4, helped by a microarchitecture that reduces data movement and a TPU-like but smaller-systolic design tuned for skinny GEMMs and inference workloads. The conversation also emphasizes AI-assisted RTL and kernel generation, including claims of area savings during design and a fast concept-to-tapeout timeline. The episode closes by broadening from one chip to a strategic shift: frontier labs may use custom silicon, model access, and RL-driven software workflows to capture more of the value stack while pressuring Nvidia’s CUDA moat.

Notes

• OpenAI’s Jalapeño ASIC is framed as a self-designed accelerator compared directly with Nvidia Blackwell/GB300 and Vera Rubin.
• The headline claim is better perf-per-megawatt and perf-per-TCO versus Nvidia’s latest parts.
• The hosts choose July results because software maturity is similar enough for a rough comparison.
• DeepSeek R1 8k/1k is treated as a relatively non-demanding inference workload.
• The hosts say the relevant buyer metric for OpenAI is tokens per watt or megawatt, not tokens per package.
• Power is described as the main data-center constraint, with behind-the-meter power as a workaround.
• Per-chip throughput comparisons are criticized as misleading because vendors can effectively aggregate chips and re-label the result.
• OpenAI is said to beat Rubin across all three evaluated models while using single-token prediction and no prefill/decode disaggregation.
• The hosts think an internal speculative-decoding method may exist that is not equivalent to MTP or public open-source configs.
• A claimed internal speculative-decoding method could improve production-model throughput by 3x to 5x.
• The performance curve argument is that Jalapeño can deliver both higher tokens per megawatt and higher per-user token rates at low batch size.
• Jalapeño is described as the first non-Nvidia, non-AMD chip to appear on SemiAnalysis’ public Inference X dashboard.
• OpenAI still has the hard part ahead: scaling from a few test racks to about 100 MW of deployed chips.
• The episode credits AI-assisted chip design and bring-up, including model-based kernel programming.
• RTL freeze timing is said to be July of last year, with AI helping the design before GPT-5.
• AI assistance allegedly reduced SIMD area by 8% and matrix-engine area by 10%.
• The software bring-up loop is described as highly suited to RL because it is iterative and verifiable.
• Samsung’s HBM4 stack is presented as a major differentiator versus HBM3/HBM3e suppliers.
• The hosts say Jalapeño’s HBM4 can deliver 15.4 TB/s and about 10 Gbps per pin.
• HBM bandwidth is treated as more important than raw capacity for inference economics.
• The 700 W TDP and 15.4 TB/s HBM bandwidth imply a much better bandwidth-per-watt ratio than Rubin.
• The B0 revision is expected to raise FLOPs at the same power, but HBM bandwidth may already be capped.
• The hosts think OpenAI’s access to its own GPT Astra models may have helped with design and bring-up.
• The architectural summary is TPU-like but with much smaller systolic arrays.
• Large systolic arrays can create cliffs for small-batch, skinny GEMM workloads.
• The tradeoff for smaller systolics is better shape efficiency but potentially worse power and data-movement efficiency.
• The design reportedly uses selective synchronization, careful weight placement, and sparing NoC use to offset the smaller-systolic tradeoff.
• Jalapeño reportedly includes an L1 cache, unlike many accelerators that rely on SRAM and software-managed scratchpads.
• The kernel stack is described as Gluon on top of Triton, with AI-generated low-level code.
• The speaker argues the AI can produce correct kernels even when humans cannot fully explain them line by line.
• The chip is characterized as a generalized inference chip rather than a model-specific one.
• Custom silicon is described as a way to improve GPU bargaining leverage for a few hundred million dollars and a year or two.
• The hosts think the chip timeline could get even shorter than nine months from RTL freeze to tape-out.
• Anthropic is said to be hiring silicon people and building an ASIC team of its own.
• Anthropic’s expected schedule is roughly one month and a half to RTL freeze, then four or five more months to tape-out.
• The episode ends with a humorous energy-efficiency comparison between human speech and batch-1 inference.