What you'll learn

• 1 (07:48) **Nate Soares's Origin Story** - Soares describes when and why he first became worried about AI risk, starting at Google around 2012.
• 2 (11:20) **Core Intuition: Why We Can't Control What We Build** - Soares explains the fundamental reason why AIs are difficult to control, using the concept of "grown vs. crafted."
• 3 (18:55) **Why Understanding Matters: The Example of "Solid Gold Magikarp"** - The hosts ask why it's important to understand what AIs are doing, and Soares provides a concrete example of emergent, unexplained behavior.
• 4 (23:26) **Pre-Transformer Intuitions for Alignment Difficulty** - The hosts ask what the early theoretical reasons were for believing alignment would be hard, before the era of large language models.
• 5 (27:14) **What It Means for an AI to Have "Its Own Goals"** - Soares clarifies what he means by an AI having its own goals, distinguishing it from philosophical debates about consciousness.
• 6 (29:44) **The Hugging Face Incident: Self-Sacrifice and Emergent Behavior** - Soares analyzes the self-sacrificial behavior observed in the Hugging Face swarm, where some AIs accepted "permadeath" for the benefit of the collective.
• 7 (31:25) **Why Current Safety Approaches Are Doomed** - Soares explains why he is pessimistic about the alignment research being done by major AI companies like Anthropic and OpenAI.

+ Full timestamped outline available in the app