Why Diffusion Will Win AI Inference with Inception Co-Founder and CEO Stefano Ermon

Summary

In this episode, Stefano Ermon traces his path from early generative-model research at Stanford to founding Inception to commercialize diffusion-based language models. He argues that the next battleground in AI is inference, not training: autoregressive models are sequential and memory-bound at serving time, while diffusion can generate many tokens in parallel and better utilize GPUs. Ermon says Inception’s Mercury models are already in production and compare favorably with speed-optimized frontier models, but required the company to build its own serving stack because existing tooling like vLLM and SGLang does not support diffusion LLMs. He sees voice agents and other latency-sensitive applications as the first major wedge, while longer-term advantages may also include better controllability and data efficiency. The broader thesis is that efficiency, not just raw capability, will define competitive advantage in AI.

Notes

• Ermon frames diffusion as a commercial bet on generative modeling that started after years of academic work at Stanford.
• His original mental model was generative world models for understanding structure and planning.
• Diffusion emerged from score-based generative models developed in 2019 with Yang Song.
• The company’s core thesis is that inference-time economics matter most, and diffusion is better aligned with parallel GPU work.
• Extending diffusion to text and code required new methods because words are discrete, not continuous.
• Mercury is the production manifestation of that thesis, with speed-optimized benchmark parity and faster serving.
• Inception is still small and heavily R&D-driven, with model training and inference acceleration as ongoing work.
• Speed is the immediate wedge, especially for latency-sensitive products where users will pay more for faster responses.
• Voice agents are an early customer wedge because the end-to-end pipeline is latency sensitive.
• He emphasizes that software and hardware optimization multiply each other, not substitute for each other.
• Building the product forces Inception to create defensible serving software and customer feedback loops.
• The underlying scientific lens is compression: better perplexity means the model found more structure.
• He argues that noise is universal and the open question is the best inductive bias, not whether structure exists.
• Diffusion is easier to control because steering can happen throughout generation with constraints or reward functions.
• That control could create product experiences unavailable to autoregressive models.
• The initial commercial wedge is speed, but the long-term upside may be new capabilities and better data efficiency.
• He says data efficiency matters most if future workloads become data-limited.
• The strategic value appears when data becomes the bottleneck and differentiation matters.
• He estimates a meaningful share of workloads are latency-critical.
• Serving diffusion LLMs required a different stack and substantial in-house kernel and engine work.
• Inception built its own SFT, RLHF, and routing stack and kept more of it closed for IP reasons.
• The company is still pushing the frontier, but specific model sizes and FLOPs remain undisclosed.
• He says models accelerate iteration, but human judgment still prunes the search space.
• The team is split between current product/platform work and next-generation research.
• Academia produced key diffusion and alignment ideas that later shaped industry systems.