AI Summary
5 min readEliseAI’s head of applied research, Mario Martone, spent 15 years as a theoretical physicist studying black holes and string theory. He now leads a team building conversational AI for housing and healthcare—industries where a single mistake can mean someone loses an apartment or misses critical care. In this conversation, Martone explains why the intellectual challenge of production AI may actually exceed that of theoretical physics, and why the 10% of failures that a research paper ignores becomes an enormous problem when you handle millions of real-world interactions per month.
The gap between benchmarks and reality
An AI system that works 90% of the time might look impressive in a research paper, but EliseAI runs about 60 million phone calls per year. That remaining 10% translates into an enormous number of failures—failures that generate complaints and, more importantly, have real consequences for people's housing and healthcare. Martone notes that when he reads academic papers on reliability, researchers often celebrate hitting 90% accuracy. "90%? That means for 60 million phone calls per year, an enormous amount of failures that is totally unbearable." The difference, he says, is between people who think about the problem theoretically and people who actually pay the consequences of those mistakes.
Continue reading the full summary in the app — free to try.
Read Full Summary →Free • No credit card required
Never miss an episode of a16z Show
Get every new episode summarized in your inbox — free, ~5 minutes to read.
No spam. Unsubscribe anytime.
What you'll learn
- 1 (00:00) **Why Real-World AI Is Harder Than the Benchmarks** - Mario Martone introduces the core tension: an AI system that works 90% of the time is a catastrophe when handling millions of real-world housing and healthcare conversations.
- 2 (01:44) **From String Theory to Applied AI** - Mario explains his path from 15 years studying black holes and string theory to leading applied research at EliseAI.
- 3 (02:36) **Why Essential Industries Need a Different Kind of AI** - Mario contrasts working at a frontier lab with building AI for housing and healthcare, where decisions have direct, consequential impacts on people's lives.
- 4 (04:42) **The Full-Spectrum AI Stack: When Not to Use an LLM** - Mario describes EliseAI's tech stack, which mixes cutting-edge LLMs with traditional machine learning and simple models, depending on the task.
- 5 (07:10) **The "Throw an LLM at It" Trap** - Mario warns that many new AI practitioners default to LLMs for every problem, but these models fail at tasks requiring consistent behavior or accurate probability estimation.
- 6 (08:35) **Building Evals at Scale to Catch the 10%** - EliseAI's team uses a mixture of top academic talent and top developers to build evaluation systems that can assess millions of conversations daily.
- 7 (10:09) **Managing Voice Latency at 60 Million Calls a Year** - Mario explains how they think about the voice latency problem, which spans both software and hardware constraints.
+ Full timestamped outline available in the app
Show Notes
What changes when an AI mistake doesn’t just hurt a benchmark score, but affects someone’s housing or healthcare? EliseAI Head of Applied Research Mario Martone argues that deploying AI in essential industries requires a very different standard of reliability.
Mario joins MTS to explain how EliseAI handles roughly 5 million phone calls a month, why LLMs aren’t the right solution for every problem, and how the team combines language models, traditional machine learning, neural networks, and internally built evaluations to decide what actually works in production.
Drawing on 15 years studying theoretical physics, Mario also explains how the scientific method shapes EliseAI’s research culture, from testing new models against its own benchmarks to fine-tuning open-source models and automating increasingly complex workflows across housing and healthcare. And he makes the case that applying AI to the real world can produce problems every bit as intellectually difficult as the ones he encountered studying string theory.
Resources:
Learn more about EliseAI: https://www.eliseai.com/
Follow Mario Martone on LinkedIn: https://www.linkedin.com/in/mcmartone/
Follow MTS: https://www.mts.now/
Stay Updated:
Find a16z on YouTube: YouTube
Find a16z on X
Find a16z on LinkedIn
Listen to the a16z Show on Spotify
Listen to the a16z Show on Apple Podcasts
Follow our host: https://twitter.com/eriktorenberg
Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures.
Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.