OpenAI’s Jalapeño! Feeling Hot Hot Hot!

Summary

This episode is a technical first reaction to OpenAI’s Jalapeño reveal at Hot Chips. The hosts argue that OpenAI optimized for user-facing latency and energy per request, not just peak throughput, and that the right way to judge the chip is on Pareto curves and delivered performance per watt. They highlight the architecture’s NUMA-like local HBM slices, the focus on KV-cache locality, and a scale-up system built around Broadcom networking to support 128-chip and 2048-chip domains. A major theme is that AI-assisted EDA may have helped OpenAI go from first RTL to tapeout in about nine months, which they see as a wake-up call for the rest of the accelerator industry. They also debate whether OpenAI should monetize the chip externally or keep it internal as a strategic advantage.

Notes

• OpenAI announced Jalapeño at Hot Chips, and the hosts wanted an immediate technical reaction.
• They frame the chip as an inference accelerator, not a training chip.
• OpenAI’s main design goals were time to last token and energy per request.
• The hosts think OpenAI is optimizing from the perspective of a model lab close to end users.
• Tokens per joule is treated as the central economic metric.
• OpenAI compared itself against SemiAnalysis Inference X in a public, power-normalized way.
• They worry that the benchmark inputs may be too narrow and possibly cherry-picked.
• The hosts look for workloads that would break the chip, especially at longer context lengths.
• OpenAI showed the chip running multiple external models, not just OpenAI workloads.
• The chip is meant to be generalized for LLM inference across small, medium, and large models.
• The hosts debate whether OpenAI should commercialize the chip or keep it internal.
• Vikram argues the core advantage is chip specialization for OpenAI’s own architecture.
• The hardest design problem was choosing what to cut from the chip.
• The tapeout happened about nine months after first RTL, which they treat as extraordinarily fast.
• They say the first chip is already on a competitive Pareto frontier against GPUs.
• A small-model chart reportedly showed more than 1000 tokens per second per user.
• The main benefit is token throughput per watt, with a 700 W TDP.
• Broadcom and Celestica are called out as important partners.
• The chip reportedly uses an x86 CPU, TSMC N3-class silicon, and possibly Samsung HBM4.
• Broadcom may be involved in interface IP, I/O chiplets, and system networking, while much of the RTL is fresh.
• The discussion centers on KV cache locality and argues against shuttling large KV states around.
• They estimate rack-scale aggregate HBM bandwidth at about 1 petabyte per second.
• The hosts say OpenAI is not fully using even current HBM capability.
• The bottleneck is late-arriving operands, not bandwidth in the abstract.
• Their solution is local HBM slices and dedicated low-latency buses.
• They explain NUMA as dedicating a slice of HBM to each accelerator to eliminate contention.
• The system scales to 128 chips in a large domain and 2048 chips across 16 racks.
• The network is described as ESUN-based and a half-flattened two-level claw topology.
• They endorse the idea that dark silicon is cheaper than idle accelerators.
• Speculative decoding is presented as draft-and-verify, with model-dependent token splits.
• They argue one-chip generality can lengthen the useful life of the accelerator.
• They think AI tools and the right team made the fast tapeout possible.
• OpenAI used AI in the design flow, including RTL generation.
• They view the result as public proof that AI-enabled EDA can accelerate chip design.
• They think AI lowers the barrier to entry for chip design and could increase demand for EDA.