Building the Automated AGI Lab: Core Automation's Jerry Tworek and Rohan Anil

July 29, 2026

AI Summary

5 min read

In 2024, Jerry Tworek was inside OpenAI scaling reinforcement learning, convinced that 2025 would be the year AGI arrived. Then he watched model after model improve on benchmarks without solving real-world tasks. "We still have work," he realized. That gap—between what transformers excel at in the lab and what they fail to do in deployment—is the founding premise of Core Automation, the San Francisco lab he co-founded with Rohan Anil. Their diagnosis: the architecture itself is the bottleneck.

Why the transformer is both marvel and trap

The transformer's genius is economic. It trains cheaply enough that the revenue it generates exceeds the cost of training it—a property not guaranteed by any architecture. LSTMs could theoretically be scaled too, but they would have been too expensive to justify the investment. The transformer made scaling viable, and scaling made the AI industry.

Continue reading the full summary in the app — free to try.

Read Full Summary →

Free • No credit card required

Listen to Audio Summary Open in App

Never miss an episode of Training Data

Get every new episode summarized in your inbox — free, ~5 minutes to read.

No spam. Unsubscribe anytime.

What you'll learn

1 (00:00) **Learning from Experience: Football vs. Mathematics** - Jerry opens with an analogy contrasting two types of learning: reinforcement-like trial and error (football) vs. deep conceptual understanding (mathematics), arguing that RL is not the only path forward.
2 (01:00) **Introducing Core Automation's Founders** - Host Sonia welcomes Jerry Tworek (ex-OpenAI, reasoning/RL lead) and Rohan Anil (ex-Google Brain, pre-training lead for Gemini), framing their unique combined expertise.
3 (01:55) **The Transformer Eulogy: Appreciating Its Strengths to Find Its Weaknesses** - Jerry explains his tweet: to replace transformers, you must first deeply understand what they do well, so you can focus on their real weaknesses rather than just making them cheaper.
4 (05:05) **The Architecture is the Issue: The Lab-to-Real-World Gap** - Jerry identifies the fundamental problem: models are trained in the lab on clean benchmarks but fail on the "murkier" distribution of real-world tasks.
5 (07:43) **Why Transformers Can't Learn Continually: In-Context vs. Fine-Tuning** - Jerry diagnoses the two existing learning mechanisms for transformers, both of which are insufficient for real-world adaptation.
6 (09:50) **The Economic Miracle of Transformers vs. Alternatives** - Jerry argues that the transformer's success was as much about economics as architecture; it was cheap enough to train that it generated more revenue than it cost, which LSTMs could not.
7 (12:49) **Why Start a Company? The Market Timing and Lab Inertia** - Jerry explains why this work is better suited to a startup than a big lab.

+ Full timestamped outline available in the app

Show Notes

Jerry Tworek led reasoning at OpenAI, convinced that scaling reinforcement learning was the path to AGI. Rohan Anil co-led Gemini pre-training and built the Shampoo optimizer. Now they've teamed up at Core Automation on a contrarian premise: the transformer has carried us as far as it can, and the bottleneck to smarter systems is no longer scale — it's the architecture itself. The missing capability is continual learning, models that adapt at test time, which transformers can't do. In-context learning taps out fast (Codex needs compacting after ~20 minutes) and fine-tuning invites catastrophic forgetting. Rohan argues pre-training and RL should be optimized end-to-end, and that transformers spend computation inefficiently. They lay out why the largest labs won't chase alternatives while locked in the coding-agent race, and why building the world's most automated lab starts with automating kernel generation—the one place frontier models still lose to a high-taste human.

Hosted by Sonya Huang and Pat Grady, Sequoia Capital