Why Hardware-Software Co-Design Is AI's Real 100x: Dylan Patel of SemiAnalysis

Summary

Dylan Patel of SemiAnalysis argues that the real AI advantage comes from co-designing the model, software stack, and silicon together, where small gains compound into step-function improvements. He uses DeepSeek, OpenAI, Anthropic, Nvidia GPUs, and Google TPUs to show that hardware choice depends heavily on architecture details like sparsity, expert structure, attention, memory bandwidth, and network topology. Patel also describes InferenceX, a continuously updated public benchmarking system with more than $50M of donated hardware, built because static benchmarks go stale as models and runtimes change weekly. He says inference will become one of the largest markets in the world, that AI compute demand is still outrunning supply, and that neo-clouds, hyperscalers, and chip vendors are all being shaped by gigawatt-scale buildouts and ecosystem strategy.

Notes

• SemiAnalysis mixes engineers and ex-hedge-fund people, creating a built-in tension between technical merit and cost analysis.
• Patel ties his hardware obsession to growing up in a motel and gas-station family business.
• He says fixing an Xbox 360 red-ring failure by shorting a temperature sensor pushed him into hardware.
• By age 12, he was moderating Android, Apple, Google, and hardware forums while tracking Intel, Nvidia, and AMD.
• He says he was Grandmaster on the North American StarCraft 2 ladder.
• After being doxxed, he shifted to real-name blogging, and SemiAnalysis traction took off.
• He says he traveled the U.S. in a truck-and-tent setup while reading semiconductor and AI textbooks and audiobooks.
• He says he attends 40-plus conferences a year across the supply chain.
• Deep technical conferences can take multiple passes to understand.
• Conference conversations reveal real supply-chain dependencies that papers do not show.
• A burned-down factory in the 1980s reportedly doubled or tripled memory prices.
• He says AI inference will be one of the biggest markets in the world, bigger than oil.
• Static inference benchmarks go stale quickly because models and runtimes change constantly.
• InferenceX got support from major clouds and model companies to contribute compute.
• The benchmark fleet now has over $50M of donated hardware and may exceed $100M with TPU and Trainium added.
• Public configs let InferenceX expose a Pareto-optimal frontier instead of cherry-picked points.
• He frames AI infrastructure around the throughput-versus-interactivity curve, then extends it to cost and power tradeoffs.
• He gives a concrete batching example of 100 users at 10 tokens/sec each versus one user at 500 tokens/sec.
• He expects space data centers to matter only on a long horizon, not in the next 3-5 years.
• He says OpenAI and Anthropic could exceed 100 gigawatts of combined compute by 2030.
• He says intelligence per watt has improved about 40x annually in recent years.
• The central thesis is that co-design across all layers can turn 2x plus 2x into 100x.
• He says DeepSeek’s expert shapes were optimized for Hopper and Blackwell, while TPUs can be poor fits for DeepSeek-style workloads.
• OpenAI did not disclose the sparsity and shape details of GPT-4o, which he says was roughly the same size as DeepSeek v3.
• He says memory bandwidth and capacity have improved slowly and remain a major bottleneck.
• He gives chip power-density figures and says Nvidia Rubin may hit about 2000 W, with Rubin Ultra around 4000 W.
• He even suggests reverse-using millions of diesel engines as generators as a power workaround.
• He argues the West underinvested in semiconductors because ads were more lucrative.
• He says there is no simple GPU-versus-TPU winner; the answer depends on co-design.
• He says the best chip depends on matrix-multiply shape, attention structure, and expert structure.
• OpenAI’s models are more sparse, while Anthropic’s are somewhat denser.
• He contrasts Nvidia’s 72-GPU NVLink switch topology with Google’s switchless ICI that can connect 8,000 chips.
• He says the CUDA moat is partly eroding because models can now code well enough to commoditize software.
• He argues the real moat is often model ecosystem design, not CUDA itself.
• Big labs can fork PyTorch and build custom stacks, while smaller teams still need open-source tooling.
• He thinks Cerebras is strong for very fast inference, which he says is a big market.
• But very large models and million-token contexts may stress SRAM-based chips like Cerebras and Grok.
• He says token volume matters less than dollars and revenue.
• He says AI ROI denial and claims that model progress has stalled are especially frustrating.
• A chip can involve roughly a thousand process steps, and even small chemical inputs can be huge businesses.
• He is excited about space data centers, asteroid mining, and co-packaged optics on a 10-year horizon.
• He says some teams are trying to innovate at silicon, software abstraction, and model layers at once, including analog compute and energy-based models.
• Everyone will keep building their own chips, and supply-chain diversification will create more bifurcation.
• He says major players will spend billions to tens of billions on custom ASIC programs.
• The market is large enough that niches can still be profitable even if Nvidia, TPU, and Trainium take most of the pie.
• He says data-center deployment is accelerating, with about 20 GW this year and more than 30 GW next year.
• Useful AI work is expanding faster than compute supply, keeping the crunch alive.
• He says model providers can rent expensive compute and still preserve gross margins because tokens monetize immediately.
• Data-center colocation power used to be about $60 per kW-month and now often trades at $120-160, with some deals reaching $200.
• Google can overbuild hardware into a 1 GW data center and use power smoothing plus utility deals to draw 2 GW briefly.
• A gigawatt allocated to Anthropic, in his view, produces more revenue than a gigawatt allocated to OpenAI.
• CoreWeave’s GPU compute is better than hyperscaler offerings, but balance-sheet structure and timing affect revenue per megawatt.
• AWS’s traditional cloud advantages can hurt AI cloud performance, especially Nitro NICs and some isolation features.
• AI customers usually rent whole GPUs or whole racks under long-term contracts, so fine-grained tenant isolation matters less.
• He says Microsoft’s data-center execution broke down when forecasts doubled, pushing it to buy neo-cloud capacity.
• Jensen Huang, according to Patel, wants a multipolar AI world rather than hyperscaler dominance.
• Neo-clouds and neo-labs are a Wild West; many will fail, but some will emerge as strong teams like Crusoe and CoreWeave.
• He says Thinking Machines has a few hundred million dollars of ARR despite being a very new product.