Will inference move to the edge?

December 18, 2025

AI Summary

5 min read

🎙️ The Voices & The Context

The Format: This interview-style podcast dives into the technical future of AI compute distribution, linking energy demands in the power sector to evolving AI infrastructure trends through expert analysis. Technical and predictive.
The Format: An interview between host and guest expert.
The Key Players:
- Guest: Dr. Ben Lee – Professor of electrical engineering and computer science at the University of Pennsylvania and visiting researcher at Google, offering deep insights into AI hardware, edge computing, and energy efficiency.

🗝️ Key Themes & Topics

The episode explores AI compute's evolution amid surging electricity demands, focusing on how inference workloads might decentralize from massive data centers to edge and on-device processing, with profound grid implications.

Topic 1: Compute Categories (Hyperscale Cloud, Edge, On-Device): Dr. Lee defines three tiers—massive hyperscale data centers (e.g., Google, AWS), intermediate edge computing for lower latency in regional facilities, and consumer devices like phones—contrasting their efficiency, with cloud dominating due to superior PUE (power usage effectiveness) near 1.1 and hardware sharing.
Topic 2: AI Workloads – Training vs. Inference: Training remains centralized in gigawatt-scale data centers for massive GPU coordination on trillion-parameter

Continue reading the full summary in the app — free to try.

Read Full Summary →

Free • No credit card required

Listen to Audio Summary Open in App

What you'll learn

1 **(00:00) 🎙️ Introduction: Dr. Ben Lee**
2 **(05:46) Defining Compute Categories**
3 **(07:47) Why Cloud Dominates Today**
4 **(09:50) AI Training Workloads**
5 **(11:13) Training vs. Inference Split Today**
6 **(13:05) Edge Inference: Arguments For**
7 **(17:08) Technical Trade-offs: Training vs. Inference**

+ Full timestamped outline available in the app

Show Notes

Today virtually all AI compute takes place in centralized data centers, driving the demand for massive power infrastructure.

But as workloads shift from training to inference, and AI applications become more latency-sensitive (autonomous vehicles, anyone?), there‘s another pathway: migrating a portion of inference from centralized computing to the edge. Instead of a gigawatt-scale data center in a remote location, we might see a fleet of smaller data centers clustered around an urban core. Some inference might even shift to our devices.

So how likely is a shift like this, and what would need to happen for it to substantially reshape AI power?

In this episode, Shayle talks to Dr. Ben Lee, a professor of electrical engineering and computer science at the University of Pennsylvania, as well as a visiting researcher at Google. Shayle and Ben cover topics like:

The three main categories of compute: hyperscale, edge, and on-device
Why training is unlikely to move from hyperscale
The low latency demands of new applications like autonomous vehicles
How generative AI is training us to tolerate longer latencies
Why distributed inference doesn‘t face the same technical challenges as distributed training
Why consumer devices may limit model capability

Resources:

ACM SIGMETRICS Performance Evaluation Review: A Case Study of Environmental Footprints for Generative AI Inference: Cloud versus Edge
Internet of Things and Cyber-Physical Systems: Edge AI: A survey

Credits: Hosted by Shayle Kann. Produced and edited by Daniel Woldorff. Original music and engineering by Sean Marquand. Stephen Lacey is our executive editor.

Catalyst is brought to you by EnergyHub. EnergyHub helps utilities build next-generation virtual power plants that unlock reliable flexibility at every level of the grid. See how EnergyHub helps unlock the power of flexibility at scale, and deliver more value through cross-DER dispatch with their leading Edge DERMS platform, by visiting energyhub.com.

Catalyst is brought to you by Bloom Energy. AI data centers can’t wait years for grid power—and with Bloom Energy’s fuel cells, they don’t have to. Bloom Energy delivers affordable, always-on, ultra-reliable onsite power, built for chipmakers, hyperscalers, and data center leaders looking to power their operations at AI speed. Learn more by visiting⁠ ⁠⁠BloomEnergy.com⁠.