The Inference Engineering Masterclass — Philip Kiely & Ali Taha, Baseten
August 3, 2026
AI Summary
5 min readThe Inference Engineering Masterclass
When Philip Kiely and Ali Taha from Baseten describe what happens when you send a 200,000-token query into their inference system, the first question isn't about GPU compute—it's about caching. "Have you sent me this query before, or at least part of it?" Kiely asks. If you have, the system routes your request to an instance where some of the prefill work can be skipped entirely, using a cached KV cache. If not, disaggregated prefill and decode sends the input to one set of GPUs that generates the KV cache and first token, then passes that to a separate set of GPUs for iterative decode—likely with a speculative decoder in front, trained on coding traffic, hoping for high draft token acceptance. It's a systems-level orchestration problem disguised as a simple API call.
The Anatomy of Supporting a New Model
When a new open model like GLM 5.2 or Kimi K3 drops, the public sees a race: one provider hits 90 tokens per second, another hits 150 within days. Behind that competition is substantial engineering work. Getting a model to produce tokens at all is relatively easy—open-source inference engines like vLLM and SGLang often get PRs merged by the model's maintainers. But production readiness requires proprietary work on top.
Continue reading the full summary in the app — free to try.
Read Full Summary →Free • No credit card required
Never miss an episode of Latent Space: The AI Engineer Podcast
Get every new episode summarized in your inbox — free, ~5 minutes to read.
No spam. Unsubscribe anytime.
What you'll learn
- 1 (00:03) **Intro & Banter** - The hosts welcome Philip Kiely and Ali Taha from Baseten, with playful discussion about Ali's "Waterloo Intern" handle.
- 2 (01:10) **What Happens When You Send a 200k Token Query?** - A walkthrough of Baseten's inference pipeline for long inputs.
- 3 (03:55) **Per-Token APIs vs. Dedicated Deployments** - When to switch from serverless to renting by the box.
- 4 (05:41) **Tool Calling: The Real Challenges** - Beyond JSON parsing, the main difficulties are on the training and output structuring side.
- 5 (09:00) **The "Inference War" & Supporting a New Model** - The intense work behind launching a new open model like GLM-5.2 or Kimi K2 on a production API.
- 6 (14:55) **Frankensteining Models: Retrofitting Vision** - How Baseten added Kimi K2 vision to GLM-5.2 without retraining the base model.
- 7 (19:13) **Layer Swapping for Efficiency** - Replacing inefficient attention layers (e.g., full attention) with efficient ones (e.g., GQA) from other models to boost inference speed.
+ Full timestamped outline available in the app
Show Notes
We first covered Baseten last year when DeepSeek mania was at peak hype. Now they have raised a monster $13B round and become one of the new cohort of AI Infra decacorns that are (with Nvidia, Intel, and the semis complex) chief beneficiaries of the Inference Inflection.
We return to Baseten at the peak of the 2026 edition of Open Weights debate. Ali has published a viral breakdown of Kimi K3:
And since you last saw him, Philip has spoken at AI Engineer and written the definitive book on Inference Engineering spotted all over SF:
Three years ago, inference engineering barely existed as a category.
Today, it is one of the most critical disciplines in AI. Inference engineering inherently tackles a different question than standard model training: “How do you turn those weights from training into a product that is fast, reliable, and affordable at scale?” Focusing on these creates an entirely new optimization problem.
In one recent GLM-5.2 experiment, quantizing more of the model actually preserved its benchmark quality while increasing throughput by 20%, because the errors introduced in different layers could cancel each other out.
Inference is no longer just the final step after training. It is becoming its own engineering discipline, with its own research problems, infrastructure, and increasingly specialized roles.
In this episode, Baseten’s Philip Kiely and Ali Taha join swyx and Vibhu to explain what actually happens after a new open model is released and what it takes to turn “we generated a token” into a fast, reliable, production-ready API.
We go deep on cache-aware routing, disaggregated prefill and decode, quantization, speculative decoding, KV-cache movement, model parallelism, GPU kernels, and the race to make frontier models up to 10× faster. Philip and Ali explain why inference optimizations can still produce gains of 20%, 100%, or even 200%; how quantization errors can cancel one another out; why identical weights can behave differently across clusters; and how Baseten grafted a Kimi vision encoder onto GLM-5.2 without changing the underlying language
More from this podcast
Latent Space: The AI Engineer Podcast →