The AI Daily Brief: Artificial Intelligence News and Analysis

The Best Way to Test New AI Models

October 7, 2026

AI Summary

5 min read

The Best Way to Test New AI Models

Every time a new AI model drops—and they're now arriving roughly every 11 days at the frontier—it arrives with a table of benchmark scores. But those benchmarks, as Nathaniel Whittemore (NLW) puts it, are "now in the training data sets" and have "limited value." The longer a benchmark has been around, the less it tells you about how a model will actually perform on your specific work. On top of that, model quality is increasingly about subjective feel and fit within your personal stack, not raw capability scores. "There are going to be times where the thing that is technically state of the art on some benchmark is worse at the version of the thing that you're doing that's lower on the charts," NLW says. This is especially true for subjective tasks like writing.

The solution, laid out by Nufar Gaspar in this practical webinar, is to build your own personal AI benchmark—a repeatable system for testing whether a new model matters for your actual use cases.

The Five-Step Benchmark Process

Continue reading the full summary in the app — free to try.

Read Full Summary →

Free • No credit card required

Listen to Audio Summary Open in App

Never miss an episode of The AI Daily Brief: Artificial Intelligence News and Analysis

Get every new episode summarized in your inbox — free, ~5 minutes to read.

No spam. Unsubscribe anytime.

What you'll learn

1 (00:00) **Why Published Benchmarks Are Not Enough** - The host explains that new models arrive every ~11 days with benchmark tables, but those scores are unreliable because benchmarks are often in training data and don't reveal how a model "feels" for your specific use cases.
2 (04:28) **The Five-Step Personal Benchmark Process Overview** - Nufar introduces the structured five-step method for building and running your own AI model evaluation, which will be the framework for the rest of the episode.
3 (07:00) **The Problem with Early Model Hype Cycles** - Nufar critiques the pattern of model releases: announcement, benchmark table, hot takes, influencer toy demos, then a week later real-world reports emerge — and why you shouldn't trust the early buzz.
4 (09:06) **Two Common Objections to Personal Benchmarking (and Why They're Wrong)** - Nufar addresses pushback: (1) "Everything is so good now, why bother?" and (2) "My company limits my model choice" — and explains why benchmarking still matters in both cases.
5 (11:27) **The Most Consequential Step: Selecting Your Use Cases** - Nufar explains how to choose 4-6 concrete, diverse tasks for your benchmark that reflect your actual work, including at least one "wishlist" task you've never gotten AI to do well.
6 (14:34) **How to Run the Blind Taste Test** - The practical mechanics of evaluating models: use a baseline, limit candidates to 2-4 (not 7), run each request in a fresh chat, anonymize results, and score side-by-side with both a numeric score and a one-line observation.
7 (17:43) **Scoring and Judging: Your Taste vs. AI as a Judge** - Two approaches to scoring results: your own subjective preference (most important) and using an AI judge with a rubric — plus the pitfalls of AI judging.

+ Full timestamped outline available in the app

Show Notes

In this Operator’s Cut with Nufar Gaspar learn how to test new AI models against your own tasks, compare outputs, and weigh quality, speed, and cost—a repeatable system for figuring out which models deserve a place in your work.

Brought to you by:

Granola - The AI notepad for people in back-to-back meetings. Try it free ⁠⁠⁠⁠granola.ai/brief⁠⁠⁠⁠

Harbor - Invest in the AI ecosystem. ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://www.harborcapital.com/aidaily⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠