AI Summary
5 min readThe Hidden Battle for AI's Next Frontier
In the world of artificial intelligence, a quiet but massive shift is underway. Two years ago, roughly two-thirds of all AI compute was spent on training models and one-third on inference—the process of actually running a model to generate answers. Today those numbers have flipped, and by the end of this year inference is expected to consume 80% of all AI compute. Yet the dominant chip on the market, NVIDIA's GPU, was designed for training, not inference, and achieves only 30–40% utilization when used for that purpose. This gap has created a new battleground, and a startup called Etched—founded by two 24-year-olds—has emerged as one of the most aggressive players trying to exploit it.
What Inference Actually Is
When you type a prompt into ChatGPT and hit enter, your request travels to a server rack filled with AI chips—typically NVIDIA GPUs. The chip first reads and processes your entire prompt in a phase called "pre-fill." Then it draws on memory from your conversation history and begins generating a response one token at a time in a phase called "decode." That entire process—from prompt to answer—is inference.
Continue reading the full summary in the app — free to try.
Read Full Summary →Free • No credit card required
Never miss an episode of Limitless Podcast
Get every new episode summarized in your inbox — free, ~5 minutes to read.
No spam. Unsubscribe anytime.
What you'll learn
- 1 (00:00) **The Inference Opportunity is Here** - The hosts introduce the episode's thesis: a startup called Etched has built an inference chip that beats NVIDIA on key benchmarks, signaling a major shift in AI.
- 2 (01:10) **What is Inference?** - A clear, non-technical explanation of the inference process (pre-fill and decode) that happens when a user sends a prompt to an LLM.
- 3 (02:15) **The Inference vs. Training Shift** - The hosts explain why inference demand has overtaken pre-training and why NVIDIA's dominance is a fragile position.
- 4 (03:58) **The Utilization Problem** - NVIDIA GPUs achieve only 30-40% utilization during inference, leaving massive room for improvement.
- 5 (04:58) **Etched: The Company and Its Founders** - Two 24-year-old college dropouts have built a company that threatens NVIDIA's inference dominance.
- 6 (07:58) **The System-Level Breakthrough** - Etched's key insight: they are not just building a chip, but an entire chip rack optimized for inference.
- 7 (10:50) **The ASIC Inspiration and the "All Red" Moment** - The hosts recount the dramatic story of Etched's first chip test failure and the team's resilience.
+ Full timestamped outline available in the app
Show Notes
Inference is becoming more important than pre-training in AI chips, including how pre-fill and decode work and why more compute is shifting toward serving models.
Today we walk through Etched’s ASIC system for transformer inference, its claims around efficiency and throughput, and the tradeoff between specialization and general-purpose GPUs like NVIDIA’s.
We also look at custom chip efforts from companies like OpenAI, Google, and Amazon, and argues that inference demand may keep growing as AI agents and long-running workloads expand.
------
🌌 LIMITLESS HQ ⬇️
EMAIL US: [email protected]
NEWSLETTER: https://limitlessft.substack.com/
FOLLOW ON X: https://x.com/LimitlessFT
SPOTIFY: https://open.spotify.com/show/5oV29YUL8AzzwXkxEXlRMQ
APPLE: https://podcasts.apple.com/us/podcast/limitless-podcast/id1813210890
RSS FEED: https://limitlessft.substack.com/
------
TIMESTAMPS
0:00 Inference’s New Frontier
2:14 Training Versus Inference
5:19 Etched’s Bold Bet
7:58 Building the Whole Rack
10:48 TSMC and the Hard Problems
13:29 Why Inference Matters
14:59 The Transformer Risk
17:02 OpenAI’s Jalapeno Chip
18:59 Why Accelerators Keep Winning
22:28 The Market Is Underpricing It
23:10 NVIDIA Is Still in the Game
24:56 Vertical Integration Wins
------
RESOURCES
Josh: https://x.com/JoshKale
Ejaaz: https://x.com/cryptopunk7213
------
Not financial or tax advice. See our investment disclosures here:
https://www.bankless.com/disclosures
Josh works with Anthropic as a contractor. All views expressed are his own and do not represent Anthropic, its leadership, or its affiliates. Nothing in this episode is investment advice.
More from this podcast
Limitless Podcast →