Why AI Infrastructure must evolve for Agent Experience — Akshat Bubna, Modal CTO

July 8, 2026

AI Summary

5 min read

Modal CTO Akshat Bubna joined the Latent Space podcast to explain how his company evolved from a serverless container runtime into a cloud platform purpose-built for AI inference and agent workloads. The conversation covers Modal’s origin story, the technical primitives that make it suited for bursty compute, the mechanics of speculative decoding, and why the company now thinks in terms of “agent experience” rather than just developer experience.

Modal started before the ChatGPT era. Bubna and CEO Eric Zhang initially set out to build a better runtime for bursty, compute-heavy workloads — the kind that Kubernetes handles poorly because it was designed for slow-scaling web servers. The insight was that many companies needed to scale up and down rapidly, use specialized hardware like GPUs, and run custom container images. Modal added GPU support a year before ChatGPT launched, but the team didn’t yet anticipate how large the inference market would become.

Continue reading the full summary in the app — free to try.

Read Full Summary →

Free • No credit card required

Listen to Audio Summary Open in App

Never miss an episode of Latent Space: The AI Engineer Podcast

Get every new episode summarized in your inbox — free, ~5 minutes to read.

No spam. Unsubscribe anytime.

What you'll learn

1 (00:39) **Origin Story: From Workflow Orchestration to Serverless Containers** - Akshat recounts meeting CEO Eric and the initial vision for a better runtime, moving away from Kubernetes complexity.
2 (02:28) **The Core Problem: Kubernetes is Not Built for Bursty Compute** - The key pain point that sparked Modal's creation.
3 (03:31) **The DX Philosophy: Self-Provisioning Infrastructure in Code** - The "software-defined infrastructure" approach, where infrastructure requirements are co-located with code via decorators.
4 (04:54) **The Shift from Developer Experience (DX) to Agent Experience (AX)** - Modal's SDK team has pivoted to think about agent experience, arguing the same benefits of DX apply to agents.
5 (06:46) **What Modal is For: A Cloud Platform for AI Applications** - Modal covers inference, training, batch processing, and sandboxes for code execution, but not traditional web servers.
6 (09:19) **The Sandboxing Journey: An Underrated Primitive** - Modal built sandboxes in May 2023, initially for running code like small developer tools, before the agent boom.
7 (12:19) **Elastic Inference: The Core Use Case** - Modal's biggest use case is elastic inference for custom models (e.g., Suno, Runway), not standard LLM serving.

+ Full timestamped outline available in the app

Show Notes

We’ve been running a bit of an Agent Cloud series surveying all the top inference/compute/cloud providers, from Databricks to Daytona to Railway and, even further back, E2B, but we’re excited to conclude this series returning to Modal, which has just raised a monster $355M Series C.

The cloud was built for developers. But agents are now changing that.

The old infra stack was designed for a human who could read docs, reason through YAML, and understand dashboards to figure out what they need when something broke. While this was painful for developers, it worked since they could fill in missing context in their heads.

However, agents don’t have that luxury. Now in this new era of agents, everything has to be tighter.

They need a place to write code, run it, inspect the output, change the environment, debug failures, and try again. Fast iteration and feedback loops with all the necessary context are crucial for agents to operate properly. Furthermore, sandboxes are a clear representation of this shift as agents can easily spin up isolated environments. This programmatic infra even extends to research:

Two years ago, we were one of the first to cover Modal with CEO Erik Bernhardsson and Alessio designed our favorite LS thumbnail of all time:

At the time, Modal was just a teeny little company with a $17M Series A.

Today, fresh off their $355M Series C, Modal is one of the clearest examples of the agent cloud future being built in real time: a cloud platform moving past traditional web app assumptions toward the workloads AI actually creates such as elastic inference, sandboxes, GPU burst, post-training, background agents, and infrastructure that agents themselves can operate.

In this episode, Modal CTO Akshat Bubna joins swyx and Vibhu to unpack why AI applications don’t fit traditional cloud assumptions, why Kubernetes was never designed for bursty compute-heavy workloads, and why Modal is now shifting from developer experience to agent experience.

We go deep on Modal’s AI infra stack: serverless functions, decorator-based infrastructure, elastic inference for custom models, GPU snapshotting, DeFlash, speculative d