⚡ An Eval for AI Scientists That Knows the Answer Key - Ian Quigley, Leash Bio
October 26, 2026
AI Summary
5 min readAn Eval for AI Scientists That Knows the Answer Key
The idea started as a playful thought: what if you could drop AI agents into a fully measured chemical landscape, give them a budget of 100 molecules, and ask them to find the best possible binder for a target protein? Because Leash Bio had already screened millions of compounds against thousands of proteins using DNA-encoded libraries, they knew the entire answer key in advance. Every molecule's binding strength was already measured. The agents would be operating in a world where the ground truth was completely known, but they wouldn't be told what it was. They would have to explore, form hypotheses, and make decisions on a limited experimental budget — just like real medicinal chemists — and only then would their choices be scored against the hidden data. What emerged was a high-throughput evaluation environment for scientific reasoning that has already generated roughly 90,000 agent rollouts and revealed systematic differences in how models of varying capability approach the problem of deciding when to stay and when to leave.
The Structure-Activity Relationship Game
Continue reading the full summary in the app — free to try.
Read Full Summary →Free • No credit card required
Never miss an episode of Latent Space: The AI Engineer Podcast
Get every new episode summarized in your inbox — free, ~5 minutes to read.
No spam. Unsubscribe anytime.
What you'll learn
- 1 (00:04) **Introduction & Recap of Previous Episode** - Hosts Brandon and RJ welcome back Ian Quigley from Leash Bio, recapping his earlier discussion on DNA-encoded libraries and the idea of a virtual sandbox for medicinal chemistry agents.
- 2 (01:30) **The SAR Task: How Agents Search for Better Binders** - Ian explains the agent-based setup where LLMs act as medicinal chemists, generating hypotheses and testing molecules within a fixed budget.
- 3 (03:05) **Measuring Scientific Decision-Making** - The key insight: knowing the full "answer key" allows evaluation of how well agents make scientific decisions, not just their final score.
- 4 (04:30) **Why Smarter Models Abandon Bad Landscapes** - The hosts probe why frontier models are better at knowing when to quit a bad search area.
- 5 (05:25) **Transcripts Reveal Plausible Medicinal Chemistry Narratives** - Ian shares his favorite pastime: dumping agent transcripts into LLMs to analyze failure modes and reasoning quality.
- 6 (06:17) **Could This Be a Game for Humans?** - The hosts suggest turning the environment into a "medicinal chemistry video game" for human chemists.
- 7 (07:19) **Which Models Are Best?** - Ian gives a quick rundown of the current leaderboard.
+ Full timestamped outline available in the app
Guests on this episode
Show Notes
Watch the full show on YouTube: https://www.youtube.com/watch?v=vhPCo-8K78U