The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

Why Models Are AI’s Next Training Dataset with Damian Borth

July 27, 2026

AI Summary

5 min read

“What happens if we take the weights of trained neural networks as the input to train a neural network to understand these weights that we have out there much, much better?” That is the deceptively simple question Damian Borth, professor of AI and machine learning at the University of St. Gallen, has been pursuing since 2021. His research, called weights-based learning, treats the millions of parameters produced by training a model not as a final product, but as a new kind of data—a modality that can be analyzed, compressed, and even used to generate entirely new models. In this conversation with Sam Charrington, Borth walks through the evolution of the idea from a small, esoteric experiment to a framework that could one day replace pre-training itself.

Weights as a Modality, Not Just an Output

Continue reading the full summary in the app — free to try.

Read Full Summary →

Free • No credit card required

Listen to Audio Summary Open in App

Never miss an episode of The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

Get every new episode summarized in your inbox — free, ~5 minutes to read.

No spam. Unsubscribe anytime.

What you'll learn

1 Why Models Are AI's Next Training Dataset with Damian Borth
2 (00:49) **The Core Premise: Weights as Input** - Sam introduces the episode's central question: as high-quality training data becomes scarce, can trained model weights themselves become the input for the next generation of learning?
3 (02:36) **The Origin Story: Fingerprinting Neural Networks** - Damian traces the work back to 2021, starting from a practical problem: how to version or fingerprint neural networks like software.
4 (04:55) **The First Proof of Concept: Predicting Accuracy Without Test Data** - The team built an autoencoder to compress weights into a latent space, then used the encoder to predict properties of unseen networks.
5 (07:57) **The Generation Problem: Blurry Weights** - The obvious next step was using the decoder to generate new networks, but this revealed a fundamental challenge.
6 (10:40) **Scaling Up: Windowed Reconstruction and RestNet** - A key breakthrough came from treating model parameters as a sequence and reconstructing windows, detaching the sequence from the original model structure.
7 (12:55) **The Community Emerges: Weights as a New Modality** - By 2024, a community had formed around weight-based learning, with researchers independently converging on the same ideas.

+ Full timestamped outline available in the app

Show Notes

For more than a decade, AI has advanced by training ever-larger models on ever-larger datasets. But as high-quality training data becomes harder to find and pretraining grows increasingly expensive, researchers are looking for new ways to keep foundation models improving.

In this episode, Damian Borth, professor of AI and machine learning at the University of St. Gallen, argues we’ve been overlooking an important source of knowledge: the models we’ve already trained. His group’s work on weight space learning treats trained neural networks themselves as data, learning from the distilled results of millions of GPU hours of optimization rather than starting from raw data each time.

We explore what it means to build foundation models of neural networks, how knowledge can be transferred across architectures and domains, why this approach could dramatically reduce the cost of developing specialized models, and whether future AI systems may be trained on collections of existing models instead of ever-growing datasets.

🗒️ Full show notes: https://twimlai.com/go/772.