Ep. 027 - OpenAI Jalapeño: Better Than Nvidia Blackwell (Accelerators)

Summary

This episode centers on SemiAnalysis’s comparison of OpenAI’s self-designed Jalapeño accelerator against Nvidia’s Blackwell/GB300 and Vera Rubin, using throughput per megawatt and total cost of ownership as the main lenses. The speakers argue that Jalapeño is already beating Rubin’s July results on tokens per megawatt and on performance per TCO, while also doing well on low-batch latency. They repeatedly stress that modern data centers are power-limited, so efficiency per megawatt can matter more than per-chip specs, especially for a vertically integrated operator like OpenAI. The conversation also dives into HBM4 vs HBM3, speculative decoding/MTP, benchmark methodology, and the role of microarchitecture and AI-assisted kernel/chip design in turning strong paper specs into real inference performance.

Notes

• The episode is about SemiAnalysis’s article on OpenAI’s Jalapeño self-designed ASIC versus Nvidia Blackwell/GB300 and Vera Rubin.
• The main framework is throughput per megawatt and TCO, not just chip-level peak specs.
• Jalapeño is said to beat Vera Rubin’s July results on per-megawatt output and on performance per TCO.
• The comparison is framed as tentative because both systems are moving targets.
• They say comparing Jalapeño to Blackwell is unfair because Blackwell uses HBM3 while Jalapeño uses HBM4.
• Power is described as the binding data-center constraint.
• The speakers use a 100 MW example to show why better tokens per megawatt can justify higher chip cost.
• For OpenAI, tokens per megawatt matters more than tokens per package because Jalapeño is an internal system.
• They argue per-chip performance can be misleading because packaging can change the count arbitrarily.
• The TCO analysis assumes OpenAI pays $279 per hour for a GB300 rather than cloud rental pricing.
• The benchmark suite includes DeepSeek R1 8K1K, Kimi K2.5, and GPTOSS 12B.
• DeepSeek R1 8K1K is described as non-agentic random-data inference rather than the hardest workload.
• Kimi K2.5 is described as a very large model, though the parameter count is corrected during the discussion.
• GPTOSS 12B is OpenAI’s relatively small open-source model in the test set.
• They say Jalapeño is winning while using single-token prediction and without prefilled decode disaggregation.
• MTP is defined as multi-token prediction, a speculative decoding method.
• They suggest OpenAI may have an internal speculative decoding method not visible in public configs.
• The speakers claim that internal speculative decoding could yield 3–5x improvement on production models.
• The public chart is described as unusual because it shows OpenAI beating Nvidia across the curve on a real test.
• The chart’s axes are tokens per megawatt and user-facing token speed, capturing an efficiency-versus-interactivity tradeoff.
• They claim Jalapeño can double tokens per megawatt, implying roughly 2x revenue and profitability for inference providers.
• At low batch size, Jalapeño is said to reach about 700 tokens per second per user versus competitors peaking around 350.
• The conclusion is that OpenAI is winning on both fast tokens and cheap tokens.
• The episode contrasts fast-decode specialists with throughput-focused accelerator families.
• OpenAI is portrayed as having a chip that does both decode and throughput well.
• They say OpenAI was the first non-Nvidia, non-AMD chip on their public inference dashboard.
• The chip is framed as the first real threat to Nvidia’s CUDA moat.
• The speakers invite other chip owners to submit results for benchmarking and publication.
• They argue the hard part is scaling from a few test racks to 100 MW of production hardware.
• The concept-to-chip timeline is described as under two years, with under nine months to tapeout.
• AI helped both silicon design and software bring-up, including kernel programming.
• They note that Anthropic’s AMD usage is partly about AI-assisted programming helping with software-stack issues.
• OpenAI’s hardware team is said to include former TPU people.
• Broadcom and Celestica are named as system-scale partners.
• Jalapeño is described as an HBM4 chip with strong bandwidth.
• The speakers say Jalapeño has less than half of Rubin’s FP4, less HBM capacity, and less TDP.
• They claim Jalapeño is comparable to GB300 on most specs except HBM bandwidth, yet still outperforms GB300 and Rubin so far.
• The speakers attribute the outperformance to microarchitecture and software rather than raw paper specs.
• They argue theoretical FLOPS and roofline numbers are often unreachable in real workloads.
• The design philosophy is that raw HBM bandwidth is easy to buy, but hard to realize in practice.
• Memory subsystems and data movement are described as the main reasons chips fail to reach raw HBM bandwidth.
• OpenAI’s microarchitecture is said to reduce data movement so more bandwidth becomes usable.
• The speakers say FLOPS tables matter less if workloads never reach them.
• Samsung’s HBM4 is described as a step up from earlier Samsung HBM generations, using 1C DRAM and SF4 logic.
• HBM bandwidth is treated as the main value driver, while capacity mainly affects cost and dollar-per-bandwidth.
• Jalapeño is said to deliver 15.4 TB/s at 700W, giving it a very strong bandwidth-per-watt result.
• They say Jalapeño’s bandwidth per watt is roughly double Rubin’s, even versus Rubin’s 1,800W max-Q mode.
• The speakers say HBM bandwidth may already be near the limit of the design.
• OpenAI’s AI assistance is said to have reduced SIMD area by 8% and matrix engine area by 10% during a February-to-July design window.