The GPU Myth: State of AI Compute 2026 | Stephen Balaban

Summary

Stephen Balaban makes the case that AI compute is still massively underbuilt and structurally non-commoditized because delivering it requires control over land, utility power, data-center construction, HPC architecture, software orchestration, and capital markets. He says demand keeps expanding as LLMs move from assistants to code generation and “software out,” so even a 10x efficiency gain would mostly become 10x more tokens consumed. The conversation goes deep on the physical stack: direct-to-chip liquid cooling, dry coolers, NVLink inside racks, InfiniBand/Ethernet between racks, RDMA, and the economics of depreciation, utilization, and GPU usable life. Balaban also details Lambda’s origin story, from face recognition and iPhone convnet experiments to DreamScope, workstation clusters, and ultimately a vertically integrated neocloud. He closes with a forward-looking thesis on “neural software,” self-assembling software, and a future where many people may effectively need at least one GPU of compute in daily life.

Notes

• Cloud compute is framed as a highly vertically integrated service, not a commodity.
• Lambda says its differentiation spans software, construction, and financing.
• One-click cluster software is presented as a key product capability.
• Balaban says the market remains underbuilt because demand is still accelerating.
• He argues higher model efficiency mainly expands usage rather than reducing total compute demand.
• The main bottleneck is described as land powered shell plus MEP equipment.
• Modern Blackwell- and Rubin-class deployments are said to use closed direct-to-chip liquid cooling and dry coolers.
• He frames compute as an energy-to-tokens pipeline from photons or gas to intelligence.
• Depreciation is described as the biggest unit cost driver.
• Retail on-demand pricing is contrasted with wholesale five-year GPU commitments.
• A large cluster is described as 72-GPU NVLink racks tied together with non-blocking spine-leaf networking.
• Frontier inference is portrayed as distributed inference for very large models.
• Training infrastructure is said to be reusable for inference.
• He says some frontier models cannot fit on a single GPU.
• He gives rough capital costs for the AI stack at gigawatt scale.
• Lambda says it runs a broad NVIDIA stack across multiple generations.
• NVIDIA’s moat is described as CUDA, cuDNN, and NCCL.
• cuDNN is described as a highly tuned matrix multiplication engine.
• NCCL is described as topology-aware communication software for collective operations.
• Lambda offers an AI-optimized parallel file system faster than standard NFS.
• The cloud software is built mostly in-house atop commodity components and KVM/QEMU.
• A 10,000-GPU cluster requires partitioning in-band, out-of-band, and compute fabrics at the same time.
• RDMA is used for direct GPU-to-GPU and disk memory access without CPU copies.
• Lambda says it is moving toward full vertical integration across land, design, financing, construction, and offtake.
• Lambda says it does not plan to lease out new data-center space.
• The company is focused on North America, with a Seoul presence via partnerships.
• Many AI workloads are latency-insensitive; cost per token matters more.
• Governance and data-sovereignty concerns are pushing domestic AI compute.
• Compute financing is split between on-demand cloud cash flows and offtake agreements.
• Lenders are increasingly comfortable financing NVIDIA-backed assets.
• He rejects the idea that GPUs only have a three- or five-year economic life.
• The economic usable life of GPUs is said to exceed accounting depreciation schedules.
• He says a compute spot market may emerge before derivatives.
• Lambda began in 2012 as a facial-recognition startup.
• The company timed its founding with the AlexNet inflection point.
• Lambda later ran convnets on iPhone using GPU image libraries and OpenGL ES shaders.
• DreamScope used Deep Dream and style transfer, reaching about a million users.
• DreamScope drove a roughly $40,000 monthly AWS bill.
• Lambda bought about $60,000 of workstation capex to replace cloud spend.
• Lambda’s hardware business reached about a $200 million run rate.
• The cloud business is now a little under a $1B revenue run rate.
• Lambda has fully exited the hardware business.
• The original DreamScope team largely remains at Lambda, and a former member co-founded Positron.
• COVID created major shocks for a capital-intensive hardware business.
• Delighting customers is described as the core organizing principle.
• As CTO, Balaban focuses on rapid data-center deployment and high-velocity buildout.
• He cites SpaceX and Lambda as examples of high-velocity deployment organizations.
• Data-center deployment is said to depend on site selection, constraints, MEP, and construction.
• AI-first data centers are contrasted with traditional cloud regions built for many services.
• He says AI data centers can accept lower uptime than traditional cloud workloads.
• Balaban’s “neural software” thesis says AI will become the software itself.
• He says the future is the model becoming the software, not just generating code.
• In this framing, bugs become misunderstandings about prompts.
• He says multimodal networks could eventually generate every pixel and waveform.
• Balaban distinguishes vibe coding from neural software.
• Neural software is described as changes in activation space rather than code execution.
• He says prototypes of neural software already exist.
• Mass adoption of neural software is predicted in 10 to 15 years.
• He cites Tesla self-driving as an example of neural software in practice.
• Agentic workflows shift time toward tests, data gathering, code search, and security.
• He says cloud services need more traditional CPU workloads and secure environments for agents.
• Many Lambda engineers already use fully agent-driven workflows.
• Self-assembling software is defined as a 24/7 agent fleet tied to product requirements and user feedback.
• He says agents will implement features live after launch and may need human help for GPU provisioning or API access.
• A gigawatt-scale AI factory is defined as a land data center consuming 1,000 megawatts.
• He compares 1 gigawatt to New York City’s roughly 5 gigawatts.
• Balaban’s “one person, one GPU” framing echoes Apple’s “one person, one computer.”
• He argues every person in the U.S. will eventually need at least one GPU of compute.
• He says this transition will take time, like Apple’s long arc.
• Agentic workflows are said to be overhyped outside software engineering.
• He highlights CAD, CAM, FEA, and CFD as good fits for agents.
• Neural OS and self-assembling software are called underrated.