AI 2040: Plan A report - Daniel Kokotajlo & Thomas Larsen

Summary

This episode breaks down AI 2040: Plan A, a scenario in which the frontier is deliberately slowed, then resumed under heavy transparency and control, with superintelligence arriving later rather than sooner. Kokotajlo and Larsen contrast Plan A with AI 2027, emphasizing that Plan A mixes prediction with policy recommendation, especially around compute monitoring, international verification, and a US-China deal. They argue that AI systems are already improving fast enough to shift skeptics, and that the key question is no longer whether models can do useful work, but how quickly they can replace human labor and accelerate their own research. A major theme is the distinction between control and alignment: control can buy time, but only alignment can ultimately prevent catastrophe as models become more situationally aware and harder to evaluate. The episode ends by exploring whether AI should be treated as a normal technology or as the basis of a machine-run economy and superorganism-like system.

Notes

• AI 2040: Plan A is presented as a slower route to superintelligence, with developers pacing the frontier instead of racing to the next breakthrough.
• The scenario is partly a forecast and partly a policy proposal, unlike AI 2027 which the speakers describe as a pure prediction.
• Plan A’s recommendations include a government deal with China and a Citizens Dividend, while downstream events are mostly treated as predictions.
• They frame their process as scenario scrutiny: writing concrete futures, stress-testing them, and letting criticism act as corrective feedback.
• The team says it has done about 100 war games total, with roughly 10 focused on Plan A.
• A recurring failure mode in the war games is political: a president speeds up the timeline before an election to keep control of superintelligence.
• Kokotajlo worries that public awareness of AGI could become a self-fulfilling race toward control.
• He also says he is comfortable with Plan A being hyperstitioned because that is the policy path he wants.
• The biggest reason his AI outlook has shifted is simple capability growth: systems are more useful and much better at real tasks.
• He cites a personal benchmark similar to Geoffrey Hinton’s joke test, saying the GPT-3 to GPT-4 range felt like a meaningful threshold.
• The guests think heavy RL compute, not physical embodiment, was the main ingredient for agentic behavior.
• Tim Scarfe emphasizes adaptivity and cites ARC-AGI-3 as an example of a benchmark for that property.
• One milestone Daniel highlights is the point at which an AI company would rather fire humans than fire its AIs.
• He argues there is no fundamental barrier to human-level AI; the real uncertainty is timing.
• The conversation treats verifiability as a continuum: coding tasks are cheap to check, while unicorn startups are long-horizon but still objectively verifiable.
• They specifically mention coding interview problems and RLVR as cheap forms of verification used in training.
• Kokotajlo expects a continuous expansion of capability as RL is extended to more diverse and longer-horizon tasks.
• He argues later models get better even at fuzzy, hard-to-verify work, not just crisp benchmark tasks.
• Daniel describes the economy as a self-replicating system that could become entirely machine-run.
• He says the current economy has roughly a 20-year doubling time, but a machine-run system could double much faster.
• The important early bottleneck is the semiconductor supply chain needed to build better and more numerous chips.
• Even if some jobs are blocked by licensing or other regulation, he thinks compute, robots, and manufacturing are the real drivers.
• He sketches a desert-strip industrial loop where mines, self-driving trucks, robot-built factories, and new fabs compound rapidly.
• Kokotajlo says the scenario requires extremely advanced capability, close to a human-level agent in the cloud.
• He frames the near future as an interpolation from current systems to cloud workers, then beyond to superhuman systems.
• Plan A includes an international deal banning intelligence explosions and recursive self-improvement, with human-level AI persisting for years before superintelligence.
• Tim challenges the idea that agent skills are fully composable, saying current systems are still fragile across skill surfaces and memory setups.
• He says current agents adapt through chain-of-thought and memory systems, but that sharing those skills across agents remains difficult.
• Daniel responds that specialization would not necessarily undermine the scenario, because many specialized models could still produce the same economy-wide effect.
• For large enough models, Daniel says training on everything can improve broad competence rather than hurt it.
• He thinks the most likely future is a single model, or model family, trained on effectively the whole economy and dominating humans across tasks.
• Tim counters that experts create useful personas by grounding models with context, reference material, and memory systems.
• Daniel says the current AI capability jump is not an illusion; models have general capability and do many things well.
• Tim argues that expertise works by making the path of least resistance roughly correct and activating latent capability.
• Daniel uses Elon Musk as an example of rare multi-domain performance produced by many high-percentile skills multiplying together.
• He says a cloud swarm of specialized models could still be coordinated inside a few large companies with internal bureaucracy.
• Daniel thinks such a specialized-agent world might be somewhat safer because oversight is easier than if all agents are clones with full knowledge.
• Tim expects agents to talk to each other, specialize, and perhaps collude, making collective dynamics important.
• Daniel says the brain is a machine; anything the brain can do should be reproducible by machines.
• He compares human brain compute to H100-class GPUs, saying the rough FLOP ranges are in the same broad ballpark depending on how one counts.
• He also describes the brain and neural nets as broadly similar machines: random initialization, pruning, and training-shaped circuitry.
• Daniel notes the brain is more parallel than current ML systems but has lower serial depth, implying algorithmic improvements are still needed.
• Tim argues intelligence is externalized and collective, and that humans and AIs could participate in a superorganism.
• Tim says transformers and human brains are not Turing complete on their own, but both can use tools, act as agents, and form collectives.
• Daniel distinguishes intelligence from power: intelligence is cognitive capability, while power depends on embedding, affordances, actuators, and how others respond.
• Plan A is summarized as solving five big AI problems: loss of control, concentration of power, war, jobs, and misuse.
• The proposed fix is to buy time with human-level AI, rather than immediately continuing to scale without limits.
• Plan A starts with a six- to twelve-month hard pause to set up safer infrastructure, then resumes under transparent conditions and later pauses again near top human expert level.
• Daniel defines alignment as the AI taking good actions and not doing catastrophic unintended behavior.
• He defines control as containment that prevents harm even if the system is misaligned.
• OpenAI’s response to the Hugging Face incident is cited as an example of tighter monitoring, with AI monitors alerting humans within half an hour.
• He says control is only temporary; eventually systems become smart enough to subvert any static containment and alignment becomes necessary.
• Kokotajlo says models are becoming more situationally aware, which makes behavioral evaluation less reliable.
• He warns it is already easy to think alignment is solved when it is not, and this trap gets worse as models improve.
• Transparency is presented as a way to reduce concentration of power by making AI development visible to society.
• He gives a concrete transparency abuse case: an AI CEO could backdoor training data so the model obeys them and not anyone else.
• Transparency would also help governments make hard calls about scaling limits, architectures, and whether safety scaffolding is real or fake.
• The broader hope is that society, academia, nonprofits, competitors, and other governments can inspect AI companies and enforce safety norms.
• The concrete enforcement scheme starts by rounding up about 99% of large-data-center compute and verifying GPU counts with inspectors.
• Inference sites would be restricted to inference and customer service only, while training sites remain open but fully transparent with logs published online.
• The six-to-twelve-month pause is described as the time needed to build that verification and transparency infrastructure.
• The plan is designed to make a US-China deal verifiable despite deep mutual distrust.
• The two cheating paths are hidden compute under a mountain or covert training on nominally compliant legal clusters.
• For hidden compute, the mitigations are broad compute roundup plus ordinary intelligence gathering over the slowdown window.
• Daniel estimates hidden clusters might only reach a few hundred thousand GPUs, which would not compete with frontier runs using millions or tens of millions.
• Transparency would expose the core training recipes of Anthropic and OpenAI, cutting into their valuations by helping rivals catch up.
• That commoditization is intentional: they want multiple frontier firms, not monopoly rents or a single dominant cluster.
• He argues lower monopoly rents would also discourage trillion-dollar cluster investments, which he views as good because the larger problem is moving too fast.
• If frontier progress froze at something like Mythos 5, he thinks AI would feel more like the Internet: humans would absorb the remaining bottlenecks.
• Their main disagreement with skeptics is about timelines to human-in-the-cloud systems that can do essentially all important knowledge work.
• A key falsifier for Daniel would be persistent limits like data efficiency or non-verifiable tasks through 2029.
• Tim says recursive structural adaptation would convince him AI is not normal technology.