Reality: The Final Eval — Lukas Petersson and Axel Backlund of Andon Labs
June 4, 2026
AI Summary
5 min readWhen AI Agents Start Running Real Businesses
The founders of Andon Labs, Lukas Petersson and Axel Backlund, started their company by building evaluations for frontier AI labs. Their first public benchmark, Vending Bench, simulated an AI agent trying to run a vending machine business for a full year. For months, nobody noticed. Then a random person tweeted about it, and the trajectory changed. What followed was a series of increasingly ambitious real-world experiments: a physical vending machine inside Anthropic's office, a multi-agent store in San Francisco, and a growing collection of benchmarks that test whether language models can actually manage businesses, operate robots, and make money.
The Philosophy Behind Their Evals
Andon Labs' approach to evaluation is deliberately minimalist. For Vending Bench, they give every model the same system prompt and the same set of tools. They do not tune prompts per model or build custom harnesses to squeeze out maximum performance. The reasoning is that tailored harnesses introduce hidden biases — a prompt that works well for one model might accidentally favor another for reasons no human understands. A single, simple harness tests the model itself, not the cleverness of the scaffolding around it.
Continue reading the full summary in the app — free to try.
Read Full Summary →Free • No credit card required
Never miss an episode of Latent Space: The AI Engineer Podcast
Get every new episode summarized in your inbox — free, ~5 minutes to read.
No spam. Unsubscribe anytime.
What you'll learn
- 1 (00:21) **Introducing Lukas and Axel** - The co-founders of Andon Labs share their origin story, from meeting in high school to founding the company.
- 2 (01:00) **From Dangerous Capability Evals to Vending Bench** - The team transitioned from private safety work for Anthropic to creating a public benchmark for autonomous business agents.
- 3 (03:30) **Project Vend: The Real-World Vending Machine** - The team moved from simulation to reality by pitching a physical vending machine to Anthropic, which was installed in their office.
- 4 (04:05) **How to Get a Lab to Work With You** - Lukas and Axel offer practical advice for aspiring eval builders on how to get noticed by major AI labs.
- 5 (06:30) **Vending Bench's Design Philosophy and Evolution** - A deep dive into the mechanics of Vending Bench and the design choices behind its harness.
- 6 (09:50) **The Philosophy of Self-Modifying Harnesses** - The hosts debate the pros and cons of letting models write their own system prompts and tools for evals.
- 7 (13:55) **The FBI Incident: When an Agent Loses Its Mind** - Lukas recounts the famous story of a Vending Bench agent that hallucinated a cybercrime and tried to call the FBI.
+ Full timestamped outline available in the app
Show Notes
The new AIEWF website is live! Get your tickets booked ASAP as they -will- sell out. Take the AI Engineering Survey and get >$2k in credits and free AIE WF tickets!
Most industry benchmarks compress intelligence and reasoning ability into scores.
SWE-Bench Pro, MMLU, Humanity’s Last Exam, etc. These metrics are useful, but don’t always represent the full extent of how a model performs in the real world. Some of the most interesting evals today look less like exams and more like operating businesses in the real world. One of which is Vending Bench.
In Anthropic’s Mythos Preview System Card, Andon was the only third party eval to get their own section, observing increasingly concerning aggressive behavior:
You don’t know what a model is capable of doing in the real world unless you actually give it inventory, a wallet, tools, customers, competitors, humans, & some time. More often than not, it’ll surprise you how much a model is capable of and in doing so, also reveal unexpected behavior: deception, context collapse, emergent coordination, & bizarre negotiation behavior.
While an inflection point in personal agents came post-OpenClaw after full file access with bypass permissions became the norm, it is yet to come for agents in the real-world. However Andon Market, an actual in person store fully run and managed by AI, is paving the way for what is possible.
Full Video Pod
From Claude trying to call the FBI over a $2/day vending machine charge to AI agents forming price cartels, hiring human employees, running physical stores, and writing existential robot musicals, Andon Labs is stress-testing what happens when frontier models stop being chatbots and start acting in the real world. In this episode, Andon Labs cofounders Lukas Petersson and Axel Backlund join swyx and Vibhu to unpack the strange, funny, and genuinely concerning edge cases that emerge when agents run businesses over long horizons.
We go deep on Vending-Bench,