Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]

Summary

Neil Movva lays out Sail’s thesis: AI is moving from interactive chat toward background agents that run for hours or days, so the winning infrastructure optimizes cost per token, not just latency. He argues that making tokens 10x cheaper creates a new product category and that the best user experience is often “no latency” because the work happens proactively before a human asks. The conversation then goes deep into the technical stack, including GPU throughput versus latency, batching tradeoffs, tensor parallelism, SRAM versus DRAM/HBM, and why KV cache is a persistent bottleneck even for very fast chips. Movva is contrarian on market structure too: he thinks open source and capability diffusion are durable, that enterprise adoption moves slowly, and that the premium for being a few months ahead may not last forever. Strategically, Sail’s edge comes from scavenging underutilized chips and power, building heterogeneous infrastructure, and pushing every layer of the stack to lower the price of intelligence.

Notes

• Sail is framed as a token factory: an API for serving open-source and large language models at an unbeatable token price, plus long-running agent VMs called sailboxes.
• The company is built for a future where AI agents run in the background for hours or days, not in real-time chat.
• The theme is abundance: make intelligence available at a sustainable cost for almost every industry.
• Movva’s North Star is lowest cost per token in the industry, with token usage increasingly self-administered by agents.
• Open source is a tailwind because customers increasingly want to own and control the intelligence they depend on.
• He says the market has moved from low-latency inference toward persistent, long-horizon tasks.
• His ideal UX is proactive: the best latency is no latency because the work finishes overnight before the user asks.
• Test-time compute scaling makes longer agent runtimes more useful; agents can already run for about an hour on suitable tasks.
• Background workloads are structurally unbounded relative to human-in-the-loop use, and may dominate real-time workloads over time.
• Deep research over 10,000+ sources is a top use case, alongside cybersecurity.
• He characterizes cybersecurity as a proof-of-work problem, where compute is spent trying to break software before it is patched.
• Verifiable problems are where abundant inference can matter most: software, formal math, and scientific discovery.
• Sail’s engineering starts with software efficiency on Nvidia GPUs, trying to squeeze more tokens from the same chip.
• Tensor cores are specialized GPU units for matrix multiplication, and Nvidia’s early ML bet began with only a small slice of die area.
• He says Nvidia first fought for only about 5% to 10% of die area for acceleration features.
• Nvidia engineers chase the absolute frontier of hardware capability, not relative benchmarks.
• The GPU is fundamentally a throughput machine, but chatbot AI pushes the system toward latency optimization.
• Batching improves throughput but increases per-request latency because each token carries other users’ traffic through the GPU.
• NVLink enables low-latency tensor parallelism, but Sail only needs cheaper chips if they are strong on compute per dollar.
• Cerebras, Groq, and similar chips are bets on a different memory hierarchy, with much more SRAM for fast access to weights and KV cache.
• SRAM and DRAM/HBM trade off speed versus density; DRAM needs refresh every ~50 milliseconds, while HBM gives hundreds of gigabytes around the logic die.
• DRAM’s dynamic nature requires constant refresh, but it allows much higher density; Blackwell is described as having 288 GB of HBM.
• Cerebras wafer-scale designs push huge SRAM bandwidth, but they still trade off capacity and flexibility.
• Even fast hardware is constrained by KV cache growth, which is dynamic and difficult to size ahead of time.
• KV cache is per-token conversation state and can exceed the model’s weights.
• Long-context reliability is still hard because models were mostly trained on much shorter sequences.
• Off-chip memory is for capacity and on-chip memory is for speed; future systems should hybridize GPUs with specialized accelerators.
• Transformers work because they can find patterns in arbitrary sequences and reweight context dynamically via attention.
• The next data frontier is RL environments and self-improvement, not more random internet text or random human feedback.
• Kernel engineering is increasingly about modifying and fusing existing GPU kernels rather than inventing entirely new ones.
• They literally write kernels on the whiteboard, describe the dispatch strategy in natural language, and have a model help implement it.
• The business is an arbitrage on underutilized compute and power, especially where others cannot or will not compete.
• Inference demand is not speculative like past networking booms; tokens are immediately valuable and used right away.
• A distributed infrastructure strategy can tolerate small, lower-redundancy data centers because workloads can be rescheduled asynchronously.
• The biggest remaining inefficiency is memory use in attention, especially KV cache compression.
• Sail is optimizing around a scavenger strategy: buy any chip and any power source it can use, rather than compete head-on with frontier labs.
• Engineering culture prizes curiosity and love of microsecond-level performance analysis more than prior CUDA experience.
• CUDA experience is a red herring because the tooling changes too quickly to reward long resumes.
• He thinks frontier labs still pay a major premium for being 3 to 6 months ahead.
• He argues capability diffusion is impossible to stop because AI-generated artifacts already permeate the web and GitHub.
• Open source will not disappear; if one leader steps back, another will emerge.
• The product vision is abundant tokens plus personalized harnesses for every company and user, probably through in-context learning.
• Cheap tokens are the fundamental input to the abundance future, and Sail plans to push on every layer of the stack.
• He wants to use every chip, power source, and suitable piece of land in the United States.
• A core contrarian move is to sidestep HBM scarcity and push harder on offloading to flash memory.
• He wants to redesign model architecture so KV cache can be offloaded to flash at much greater scale.
• Serving at 1 to 10 tokens per second is his stated North Star.
• A trillion tokens at OpenAI pricing is at least $5 million.
• For some model sizes, a trillion tokens may already be measured in tens of thousands of dollars for a single job.
• There is always demand for intelligence; the product problem is building the on-ramp to it.
• He says customers often cannot afford enough tokens for free-tier users, and Sail wants to fix that.
• He is bullish on Nvidia short term, but says performance-per-watt gains from Hopper to Blackwell to Rubin are not dramatic.
• He makes a similar point about TSMC nodes, arguing 5nm to 2nm does not radically change performance per watt.
• Losing TSMC access would shock supply, but he thinks western fabs are not that far behind.
• Chip architecture choices like sparse versus dense attention and lower-precision data types have major downstream effects.
• Chip startups should identify the supply-chain bottlenecks, especially HBM scarcity, and attack one spiky constraint.
• Memory fab expansion is slow, and shortages will likely ripple into downstream consumer prices.
• Nvidia’s ecosystem strategy is to avoid competing with customers directly and instead cultivate many neo-cloud and inference buyers.
• He thinks Nvidia benefits from a broad ecosystem of independent buyers so no single customer can verticalize away from it.
• His mentors taught him to understand the full stack, from gate-level silicon to internet-scale services.