🔬“We have foundation models for language, not for physics” — Anima Anandkumar, Bren Professor of Computing
August 26, 2026
AI Summary
5 min readAnima Anandkumar, Bren Professor of Computing at Caltech, has spent decades working at the intersection of AI and the physical sciences. Her core argument is that the AI community has built powerful foundation models for language, but lacks equivalent models for physics. Language models can generate hypotheses, but they cannot test them in the physical world. To bridge that gap, AI must learn to simulate, design, and control physical systems with guarantees. This requires models that respect the structure of the natural world—geometry, multi-scale phenomena, and conservation laws—rather than treating everything as a fixed-resolution grid of tokens.
Why standard neural networks fail for physics
The fundamental problem, Anandkumar explains, is that conventional neural networks assume fixed-size inputs and outputs. A language model has a fixed vocabulary; a vision model works on a fixed pixel grid. But the physical world is inherently multi-scale and continuous. Weather, fluid dynamics, and material deformation all happen at infinite resolution, and a model that cannot zoom in or out is fundamentally limited.
Continue reading the full summary in the app — free to try.
Read Full Summary →Free • No credit card required
Never miss an episode of Latent Space: The AI Engineer Podcast
Get every new episode summarized in your inbox — free, ~5 minutes to read.
No spam. Unsubscribe anytime.
What you'll learn
- 1 Timestamped Navigation Outline
- 2 (00:00) **Why AI Weather Modeling Was Initially Dismissed** — Anima recounts how weather scientists cautioned that decades of physics-based modeling couldn't be beaten by AI, but neural operators proved otherwise
- 3 (01:27) **Introduction and Guest Background** — Hosts Brandon and RJ welcome Anima Anandkumar, Bren Professor at Caltech, who has spent two decades bridging AI theory and practice
- 4 (03:12) **The AI + Science Thesis** — Anima explains her core framework: AI needs guarantees to work in the physical world, not just hypothesis generation
- 5 (06:22) **TorchLean: Formal Verification for Neural Networks** — How to write neural networks in Lean to prove bounds on outputs, robustness, and safety
- 6 (11:24) **Why Physics-Informed Neural Networks (PINs) Fail** — The fundamental limitation of trying to solve PDEs from scratch without data
- 7 (14:08) **Intuition: Neural Operators vs. PINs** — The key difference is having training data that teaches the model how to solve problems across instances
+ Full timestamped outline available in the app
Show Notes
A few years ago, Caltech Prof. and co-founder of Accelerated Understanding, Anima Anandkumar set out to develop the first open-source weather model with AI. Talking to experts in the field, she was met with skepticism. Weather is chaotic, physics simulations are hard, have been developed for decades, and require supercomputers, the data just isn’t there. Despite reservations, Anima went forth and built. Within a year her team had developed FourCastNet, a predictive model that is competitive with the best physics-based simulations available. Thanks to Anima, and her follow up work, anyone can now predict weather accurately over a short timescale using consumer grade GPUs.
In the fifteen or so science episodes we’ve released on Latent.Space, we’ve covered atoms, molecules, materials, biology, and math. Anima is a pioneer in studying physical systems that are continuous. Weather, fusion, and fluid or heat flow are huge areas of science that are extremely difficult to model: they are large, chaotic, and fundamentally multi-scale. This is a field the AI community has somewhat neglected, but one we expect will grow fast. We plan to cover large physical systems more in coming episodes.
One thing you can glean from Anima’s work is that this area of AI resists the scaling ideas that have permeated the rest of the field. The data isn’t there: open source datasets in many of these domains are limited to tens or hundreds of thousands of examples, far from what token-hungry transformers need. Even worse, the resolution that physics demands pushes the context length into the hundreds of billions, so you can’t just throw more tokens at the problem. That isn’t a ceiling though, just a slower road: progress here comes from building in structure and inductive biases. Sorry for all you bitter-lesson-pilled language modelers.
“If each dimension is even a few hundred grid points, which is where industrial scale starts... we’re talking hundreds of billions to even a trillion context length. So forget ever having a transformer for anything of this scale, all of the world’s compute will not be enough.”
The math underneath
To tackle these systems, Anima pioneered a technique known as Neural Operators, one of the most beautiful theoretical developments in AI of the last decade. These allow you to combine data and physical laws to enable multi-scale inputs and outputs. We’re no longer modeling a grid, we’re modeling a function that evolves over many scales. This allows Anima and crew to build in priors based upon physical intuition.
To see how physical priors are still helpful for AI modeling, let’s revisit the problem of weather forecasting on a global scale. The earth is a sphere, which m
More from this podcast
Latent Space: The AI Engineer Podcast →