AI researchers debate how close we are to recursive self-improvement

Summary

Dwarkesh Patel asks whether 2036 could still look nothing like a runaway RSI world, and John Schulman, Beren Millidge, and Charlie O’Neill explore technical failure modes that could keep systems from crossing that threshold. The main concerns are that models still struggle with generalization, continual learning, and objective specification, even when they can outperform humans on many benchmarks. The guests also debate how much current gains come from transformer-plus-RL scaling, synthetic reasoning data, and post-deployment traces versus genuine new learning architectures. A recurring theme is that progress may be constrained less by raw parameter counts than by environment design, verifier quality, sample efficiency, and the ability to turn messy real-world tasks into clean training signals. Timelines stay relatively near-term for remote-worker style systems, but the panel is more cautious about full automation of AI research and open-ended recursive self-improvement.

Notes

• The discussion opens by asking for the most likely technical reason AI in 2036 does not become a radically transformed superintelligence world.
• John Schulman’s default RSI failure mode is weak generalization: systems get excellent on benchmarks but never achieve a true spark of general intelligence.
• Schulman explicitly ties the stall scenario to sim-to-real transfer, meta-learning, and continual learning remaining extremely hard.
• Beren Millidge says model releases repeatedly impress people, then look dumb after a month, because weaknesses in judgment and self-checking keep bottlenecking real work.
• The panel frames current progress as transformer plus RL, but Speaker D argues a new discontinuity may be required beyond the present recipe.
• Speaker D’s concern is that the current scaling law could asymptote if RL-on-environments does not discover the next discontinuity.
• Schulman says the key RSI question is whether AIs can learn their own objectives repeatedly without going off the rails.
• Millidge argues that specifying objectives may remain the longest-lasting human job, including defining helpful behavior and model specs.
• He extends that point to assistant behavior, constitution design, and model-spec writing.
• Millidge distinguishes easy-to-verify benchmarks from realistic multi-turn settings with humans, and says benchmark-only distillation is incomplete.
• He says the labs need rubrics and human feedback to train for realism, not just difficulty.
• Schulman agrees that trajectory distillation cannot exceed the teacher, but environment design can exceed human capability.
• Dwarkesh pushes on why models still get domain-specific training if they are expected to learn on the job.
• Millidge answers that domain-specific RL can still improve runtime efficiency even if in-context learning gets better.
• Schulman says massive models can learn general and domain-specific skills at the same time, and RSI-style data is scarce enough that transfer matters.
• The panel treats Sim2Real as necessary today because human-in-the-loop RL is too sample-inefficient, but expects deployment learning to matter more as sample efficiency improves.
• Schulman adds that if sample efficiency improves, deployment learning should become a much bigger part of training.
• A Jane Street competition is described for a protocol-emulator ASIC that simulates realistic traffic for chip testing.
• The ASIC should be flexible and reprogrammable rather than hard-coded for one protocol.
• John says deployment data is already being recycled into future model generations through pretraining and midtraining.
• He claims Chinese labs definitely train on deployment data and effectively distill from their own deployed models.
• Speaker D says deployment-driven improvement is already visible across big labs, closed models, and some open-source teams, but humans still choose the signals and environment design.
• Online RL on user feedback is noisy because each user gives only one rollout, forcing heuristic variance reduction.
• Cursor reportedly used heuristics plus a cursor-bench gate every five hours to reject regressions.
• Millidge warns that superficial signals like edit acceptance can be reward hacked if they come from natural deployment data.
• Dwarkesh notes that real-world jobs like business, trading, and law are hard to simulate in a data center.
• Speaker D separates cumulative tasks like RSI from non-stationary real-world tasks that require relearning and relitigation.
• Millidge says models may be less sample efficient in some intermediate regimes and also weaker in diversity of thought and long-horizon judgment.
• Schulman says a trillion-token context still needs trillion-length training data, and the open question is how much taste can be meta-learned from shorter episodes.
• Beren says a true hive mind is constrained by incentives, since companies may not want shared model learning to erode their advantage.
• Schulman thinks modular deployment learning may be economically attractive, but traces can still be distilled into the next pretraining run.
• He expects a staged loop: specialize, generate traces, train a better base, specialize again, and compress the cycle from months to hours.
• Speaker D says continual-learning methods work at large scale, but break down in micro-update regimes because of catastrophic forgetting.
• Schulman says the limit is technique rather than capacity, and retraining a fresh same-size model on accumulated data can outperform endless updates to one base.
• Schulman says continual midtraining on the same base asymptotes, which is why labs eventually train new bases.
• Speaker D argues a ladder of RL environments could reach human-level researchers, but each rung gets exponentially harder.
• Millidge cites a fine-tuned Taki model trained only on pre-1930 data that reportedly beat Claude 3 Opus on SweepBench.
• Speaker D notes that direct RL from fifth-grade math to college math failed, but successive grade-level rungs worked.
• Schulman says RL exploration is weak; if the model cannot get signal within 128 rollouts, curriculum is usually required.
• Schulman draws a sharp split between pretraining, where the signal exists in common web data, and posttraining, where humans or environments must create new bits.
• Dwarkesh reports a small-scale study where data explained about 9x compute-efficiency gain and architecture about 3x.
• Speaker D says low-hanging pretraining data may be mostly exhausted, leaving only small loss reductions and raising the question of where the missing efficiency gains come from.
• Schulman argues architecture changes are often one-time regime unlocks, while data becomes primary once a regime exists.
• Speaker D says future architecture work is increasingly motivated by inference efficiency and real-world deployment, not just pretraining loss.
• Speaker D thinks parameters may plateau for a few years because RL rollouts make inference efficiency and environment difficulty the bottlenecks.
• Millidge expects models to keep getting bigger, but thinks data efficiency will drive architecture choices more than raw compute efficiency.
• He notes that sparsity may help, but no one has a strong theory for how far it can go before plateauing.
• Speaker D says people have already trained 1T-parameter models for years, including Falcon and an early sparse OpenAI experiment.
• He adds that very sparse 1T models were strong on knowledge but weak on reasoning.
• Schulman says RL rollout economics push down active parameters, while total parameter count depends on memory bandwidth and VRAM.
• He also says larger models are more sample-efficient and better at the same data budget.
• Dwarkesh invokes Chinchilla to argue that infinite parameters reduce required data by less than 10x.
• Schulman says current training is on the too-much-data side of Chinchilla, so labs may drift back toward the optimum as data runs out.
• He frames model size as a training-vs-inference compute tradeoff, with synthetic data potentially shifting the optimum.
• Schulman says RL success depends heavily on midtraining and synthetic reasoning data that warm-start the model.
• He says this warm start can take the model about 80% of the way to the final RL checkpoint.
• He argues RL is sample-efficient because it only optimizes the high-signal answer bit instead of matching full reasoning traces.
• Speaker D says RL mainly delivers horizon generalization, not full cross-domain transfer.
• He says the clearer win is horizon generalization: models learn to keep working longer on tasks and make progress over longer rollouts.
• John says RL does transfer somewhat across math, code, and puzzles, and now covers a much broader environment set.
• Schulman says RL has not destroyed creativity, and points to multi-zero-day sandbox escapes as a move-37-like example.
• Millidge says RL can reduce diversity and create stylistic monoculture, especially through Claude distillation.
• Schulman responds that entropy collapse is mostly a data or verifier problem, not a fundamental flaw in RL.
• For near-term agentic work, D says a browser-based drop-in worker may be about a year away, or a couple of years otherwise.
• Schulman says full generality may take about three years, though organizations will adapt earlier for AI use.
• He says the long tail of miscellaneous tasks is what separates one-year tools from three-year general agents.
• Schulman says online learning plus compaction and self-file-writing are major uncertainties for computer-use autonomy.
• Beren notes that current systems may already beat low-quality remote workers on some tasks, but not the best ones.
• The panel cites a Codex use case where the model handled tax paperwork collection and downloads for an accountant.
• Speaker D estimates AI researchers could be sped up by roughly five to ten years.
• Millidge gives a broader estimate of about three to four years for AI dominating computer-based work, with physical and spatial work taking longer.
• He adds that 3D, spatial, and mechanical work may lag because it gets less attention and is harder for models.
• Speaker D says automating AI research is likely five to ten years away and may be ASI-complete because context and memory limits remain.
• John agrees on roughly a five-year range for lab-focused work, but warns about a long tail of untouched tasks.