OpenAI's Dan Roberts: Why AI Can Now Make Discoveries

Summary

Dan Roberts, who leads OpenAI’s Foundations of Reinforcement Learning work, makes the case that RL is now a core mechanism for turning compute into intelligence, especially once models are strong enough to use test-time compute to reason rather than merely autocomplete. He uses OpenAI’s recent mathematics result as an example of explorer-style search: the model assumed a conjecture was false, pursued a long contrarian chain of reasoning, and found a disproof with an unexpected cross-field connection. Roberts contrasts OpenAI’s informal-language approach with DeepMind’s Lean/formal-proof route, and explains RLHF as preference comparisons used to train a proxy reward model because humans cannot provide real-time training feedback. A recurring theme is that scientific discovery requires more than scale: research taste, problem selection, and the ability to sustain long-horizon reasoning are still missing pieces. He argues the trajectory is smooth rather than discontinuous, but expects more math, science, and engineering breakthroughs soon, with AI increasingly participating in the research loop itself.

Notes

• OpenAI’s Foundations of Reinforcement Learning team studies RL as a scientific discipline, including scaling laws and what RL can or cannot teach models.
• The team’s practical goal is to turn available compute into intelligence by intervening earlier in the model-development pipeline.
• Roberts traces his path from theoretical physics at MIT to frontier AI work.
• DeepMind’s DQN Atari and AlphaGo results helped pull him into machine learning.
• He coauthored Principles of Deep Learning Theory to borrow statistical-physics ideas for understanding neural networks.
• He joined OpenAI because frontier AI seemed to be where the most interesting scientific progress was happening.
• Roberts sees AI’s usefulness for science as a smooth progression, not a hard discontinuity.
• OpenAI’s math result involved a conjecture that was assumed true, then attacked by assuming it was false.
• The result disproved a lower-bound conjecture and found an unexpected connection to another field of mathematics.
• He distinguishes DeepMind’s formal Lean-based proof search from OpenAI’s informal-language approach.
• OpenAI’s public math work is mostly in the informal setting, using language models trained to reason at test time.
• He uses a Super Mario example to contrast pure imitation with true reinforcement learning.
• In RL, an agent acts in an environment, gets feedback, and learns from reward signals, often with curriculum structure.
• RL works because it gives feedback at the right level of abstraction, rather than forcing a learner to absorb too much at once.
• RL breaks down when feedback is sparse, delayed, or only available after many actions.
• RLHF uses human preference comparisons to train a value function that serves as a reward proxy.
• Humans cannot stay in the loop during training, so RLHF depends on a learned proxy because of latency.
• Self-play is a key RL setup because it can move from random play to expert play in zero-sum settings.
• He describes a poker-bot tournament where the objective was not to maximize winnings but to avoid losing.
• He places the OpenAI unit-distance proof in the explorer camp: contrarian, long, and broad in mathematical search.
• A later Erdős-related disproof by humans appears to have been inspired by the OpenAI result.
• Roberts says RL is no longer a minor add-on; it is central to converting compute into useful intelligence.
• The reason RL started working is that powerful pretrained models can use test-time compute to reason through tasks.
• He cites DeepSeq as an example of training on sequences filtered for correctness, which sharply limits the information per token.
• Roberts argues RL drives coding and science breakthroughs by teaching models to think at test time.
• He rejects the claim that LLMs are not intelligent and says language is the grounding layer because the internet and human knowledge are encoded in language.
• His contrarian view is that scale alone is insufficient; good ideas must steer scaling, especially when scaling RL on top of pretraining.
• He expects future good ideas may come from AI, continuing the loop between scaling and discovery.
• Test-time compute is described as token generation that acts like a scratch pad or running thought process.
• Longer test-time compute could let a model spend years on a problem like the Riemann Hypothesis.
• The model’s natural mode of thought is in language, and examples of reasoning can elicit chain-of-thought even before RL.
• The output of RL is a model that can think at test time, exposed by reasoning-effort dials.
• A verifiable reward is one that can be checked mechanically, such as exact integer matching in math.
• Creative writing is a harder RL target because taste cannot be reduced to a simple string-match reward.
• He expects OpenAI to ship products in consulting, banking, and legal domains, with some RL involved.
• He defines general intelligence as expanding the training distribution while preserving robustness to out-of-distribution cases.
• His physics lens is to reason big-to-small: understand the large system by studying simplified subsystems that preserve the phenomenon of interest.
• He rejects the idea that scaling creates mysterious discontinuities; instead, the missing ingredient should be identified in smaller models.
• He compares AI scaling analysis to theoretical physics, where simplified models are used to study complex systems like the Standard Model.
• The missing bridge in scaling laws is from microscopic weights and biases to thermodynamic-like emergent behavior.
• He thinks the next six months will bring more math and science breakthroughs and more capable models.