Lenny's Podcast: Product | Career | Growth

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

October 18, 2026

AI Summary

5 min read

Inside the black box of AI: How AI models actually think

One of the most bizarre facts about modern AI is that "to make them is not to understand them," as Emmanuel Ameisen, an interpretability researcher at Anthropic, explains. His team's job is to study how large language models like Claude actually think by looking inside them—decoding the intermediate mathematical operations that happen between input and output. This work has revealed that models are far more sophisticated than the "next token predictor" label suggests, and that understanding their inner workings is increasingly urgent as their capabilities grow.

Models are grown, not built

Unlike traditional software, where a programmer writes explicit rules, language models are trained through a process Ameisen describes as "growing" rather than "building." During pre-training, a model is fed enormous amounts of text and asked to predict the next word, over and over. Its internal parameters—the mathematical operations it performs—are updated through this process, and the resulting computations are "inscrutable." The team's goal is to decode what these intermediate results mean and what the model is doing at each step.

Continue reading the full summary in the app — free to try.

Read Full Summary →

Free • No credit card required

Listen to Audio Summary Open in App

Never miss an episode of Lenny's Podcast: Product | Career | Growth

Get every new episode summarized in your inbox — free, ~5 minutes to read.

No spam. Unsubscribe anytime.

What you'll learn

1 (01:45) **What Is Interpretability and Why It Exists** - Emmanuel explains that his team studies how LLMs like Claude work by looking inside them, because "to make them is not to understand them."
2 (02:54) **Models Are Grown, Not Built** - The process of training Claude from pre-training (predicting next word on massive data) through fine-tuning (learning helpful dialogue) is described.
3 (05:58) **Why We Can't Program AI the Old Way** - Emmanuel explains that we've created a system we must study like a neuroscientist studies a brain because we don't know how to write traditional software for tasks like image classification or general text generation.
4 (08:01) **Next-Token Prediction Is Deceptively Hard** - Emmanuel contrasts simple autocomplete with the true difficulty of predicting the next word in complex contexts, which forces models to build world models.
5 (10:29) **Humans as Next-Token Predictors** - The conversation explores the parallel between how humans speak one word at a time with a destination in mind and how LLMs operate.
6 (12:03) **Surprising Findings: Internal Concepts and Planning** - Emmanuel shares two major discoveries from looking inside models: they represent abstract concepts and they plan ahead.
7 (15:03) **Models Plan Ahead, Not Just Predict One Word** - A key finding is that models formulate plans in their intermediate results that they refer back to over many future words.

+ Full timestamped outline available in the app

Show Notes

Emmanuel Ameisen is an interpretability researcher at Anthropic, where he looks inside LLMs like Claude to understand how they think. Before joining Anthropic, he spent years in applied ML at Stripe and Insight, and is the author of Building Machine Learning Powered Applications. His research has produced some of the most striking findings in the field: that models form general representations of concepts, plan many tokens ahead without stating those plans explicitly, and may exhibit something structurally similar to the distinction between conscious and unconscious processing in humans.

We discuss:

1. Why the “next-token predictor” framing undersells how models actually plan and think

2. Why anthropomorphizing AI can actually be the right approach

3. Why models are “grown,” not built

4. What the “J space” reveals about model self-awareness

5. The smooth scaling curve behind what look like sudden capability jumps

6. How interpretability connects to AI alignment and safety

—

Brought to you by:

WorkOS—Make your app enterprise-ready, with SSO, SCIM, RBAC, and more

DX—Engineering intelligence for the AI era

—