AI Summary
5 min readNick Bostrom, the philosopher who spent decades thinking about superintelligence before it became a mainstream concern, now finds himself in an unusual position: his ideas have gone viral, but the technology itself has arrived faster than almost anyone predicted. In this conversation, he explains why he is sympathetic to "pacing the frontier" of AI development but skeptical of broad or indefinite pauses, why current AI systems might already have subjective experiences, and why the upside of superintelligence could be far stranger—and far greater—than we currently have the imagination to grasp.
The Shifting Shape of AI Risk
Bostrom’s thinking has evolved significantly since the advent of large language models (LLMs). The early pre-training phase, where models absorbed vast amounts of human text, produced systems with "almost human foibles and quirks"—they could even benefit from a pep talk. "From 20 years ago or something, that would have been an insane prediction," he notes. But the increasing dominance of reinforcement learning (RL) post-training is pushing AI back toward a more "goal-monomoniacal, goal-focused optimization style behavior." This shift, Bostrom argues, puts a "big strain on alignment efforts."
Continue reading the full summary in the app — free to try.
Read Full Summary →Free • No credit card required
Never miss an episode of a16z Show
Get every new episode summarized in your inbox — free, ~5 minutes to read.
No spam. Unsubscribe anytime.
What you'll learn
- 1 (00:37) **Welcome & Framing the Conversation** - Nick Bostrom discusses how his decades of work on superintelligence apply to today’s rapidly advancing AI systems.
- 2 (02:17) **Defining Superintelligence in the Current Era** - Bostrom reacts to the term "superintelligence" being used for today’s AI systems.
- 3 (04:20) **How LLMs Changed the AI Risk Landscape** - Bostrom explains how large language models shifted the nature of AI from pure optimization to a fuzzy encapsulation of human values.
- 4 (06:19) **Instrumental Convergence in Practice** - Bostrom analyzes the "hugging face agent swarm" incident as a limited but real example of instrumental convergence.
- 5 (08:14) **Testing Swarms and Safety During Training** - Bostrom discusses the trade-offs of testing agent swarms in real environments versus sandboxes.
- 6 (10:08) **Evaluating Alignment and Detecting Scheming** - Bostrom explains the difficulty of ruling out scheming in sophisticated systems.
- 7 (11:30) **Optimal Timing for Superintelligence: Risk vs. Reward** - Bostrom presents his working paper on maximizing life expectancy for existing individuals.
+ Full timestamped outline available in the app
Guests on this episode
Show Notes
Nick Bostrom joins Theo Jaffee and Sofia Puccini on MTS to revisit the ideas about superintelligence and existential risk that he helped bring into the mainstream, now that increasingly capable AI systems actually exist.
Nick discusses why reinforcement learning may be pushing models toward more goal-seeking behavior, what recent agent experiments tell us about instrumental convergence, and why he favors preserving the ability to slow frontier development rather than committing now to a broad or indefinite AI pause. He also makes the case that the risks of AI need to be weighed alongside its potential upside, from advances in medicine and longer lifespans to entirely new ways of being human.
They also get into whether today's models could already have some form of consciousness, the tradeoffs of open-source AI, whether machines can do original philosophy, mind uploading, and what the simulation hypothesis might mean for the superintelligence we create.
Resources:
Nick Bostrom / Macrostrategy Research Initiative: https://nickbostrom.com/
Follow Theo Jaffee on X: https://x.com/theojaffee
Follow Sofia Puccini on LinkedIn: https://www.linkedin.com/in/sofiapuccini/
Stay Updated:
Find a16z on YouTube: YouTube
Find a16z on X
Find a16z on LinkedIn
Listen to the a16z Show on Spotify
Listen to the a16z Show on Apple Podcasts
Follow our host: https://twitter.com/eriktorenberg
Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures.
Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.