Building the Automated AGI Lab: Core Automation's Jerry Tworek and Rohan Anil

Summary

Jerry Tworek and Rohan Anil make the case that the AI industry has mastered large-scale pretraining and RL, but still lacks models that can adapt continuously in the wild. Jerry argues the core gap is the mismatch between lab training and messy real-world deployment, while Rohan focuses on how current transformers spend computation inefficiently, especially in shallow, autoregressive inference. Both emphasize that architecture, optimization, and hardware kernels must be co-designed end to end, not treated as separate stages. Core Automation’s near-term wedge is automating kernel generation on Blackwell/B200 hardware, which they see as essential to faster iteration and to exploring alternative architectures. Their broader goal is an automated lab that can evaluate models by whether they improve Core Automation’s own day-to-day work over time.

Notes

• Jerry frames RL as one stage of learning-from-experience, not the endpoint.
• The episode opens with Jerry and Rohan’s background as Core Automation founders.
• Jerry says the industry has mastered large-scale pretraining and RL.
• He argues architecture is now the main bottleneck to smarter systems.
• Jerry’s central tension is lab training versus real-world deployment.
• Benchmark gains, he says, have not solved real-world tasks.
• He wants test-time learning on user data and messy real-world distributions.
• In-context learning is useful but limited in scale.
• Jerry cites about 20 minutes of Codex usage before compaction.
• Fine-tuning, he says, risks catastrophic forgetting and poor data efficiency.
• Jerry wants a meta-learned algorithm at the architectural layer.
• He argues Transformers won partly because they were cheaper to scale than LSTMs.
• He says architecture research has been too constrained by small-scale experiments.
• Rohan describes the Transformer as an efficient way to spend computation.
• He says industry compute is increasingly inference-time token generation.
• Rohan says most transformers are shallow, often around 100 layers.
• Chain-of-thought and RL increase depth, but only one token at a time.
• Jerry says transformers are capped at static knowledge and workflows trained from the internet.
• He says RL is important but not the only way humans learn from experience.
• Rohan says pretraining should be judged end to end, not just by perplexity.
• He argues architecture and optimization are two sides of the same coin.
• Jerry defines AGI as a model that can improve itself without humans in the loop.
• Core Automation’s near-term bottleneck is kernel generation on Blackwell GPUs.
• Kernel bottlenecks can block novel ideas from shipping.
• Rohan says a 60x QR-kernel speedup could open new algorithmic space.
• Core Automation says it exists to build continuously learning models.
• The company’s goal is to search architecture space faster than anyone else.
• They want systems that learn at test time and improve the lab’s daily work.
• Rohan says prior kernel work took only a few specialists and about two years.
• Rohan describes a past memory-augmented transformer idea called Ngram memory.
• He says the idea needed sparse gather/scatter hardware support that never arrived.
• Rohan says a 60x QR result came from competition winners versus cuSolver.
• Jerry says current systems still rely heavily on humans.