Red-Teaming after Mythos

Gray Swan's co-founders Zico Kolter and Matt Fredrikson started from a simple observation: large language models are software, but they have fundamentally different vulnerabilities than traditional software. "They can be tricked. Like people get tricked sometimes," Fredrikson says. Their startup, Gray Swan, exists to help enterprises understand and mitigate those risks — not by using AI to solve cybersecurity problems, but by treating the AI systems themselves as untrusted entities that need their own security layer.

The Lethal Trifecta

The core risk framework Gray Swan uses comes from Simon Willison's "lethal trifecta" of prompt injection. For a real security incident to occur, three conditions must align: the agent ingests external data from untrusted sources, it has access to private internal information, and it has the ability to exfiltrate that data. When all three are present, you have a genuine vulnerability. The key insight is that this is not a binary problem — just like traditional software, you can use AI productively despite vulnerabilities. The goal is not provable security but a better point on the usability-versus-security Pareto frontier.

The Red-Teaming Stack: Shade, Arena, and Signal