Building Observable Systems with eBPF and Linux (with Mohammed Aboullaite)

October 31, 2025

AI Summary

5 min read

The worst system monitoring setup Mohammed Aboullaite ever witnessed was in the early 2000s: a company with three servers signed a support contract worth the equivalent of $2 million today. He half-jokingly offered to camp next to the server rack for a year for half that price. They declined and went bust. The anecdote sets up the core question of the episode: how do you monitor modern distributed systems, built with multiple languages and databases, without resorting to ad hoc techniques for each component? Aboullaite's answer is to tackle the problem from the kernel level upward, using eBPF to build a unified observability strategy.

The shift from ad hoc profiling to continuous profiling

Continue reading the full summary in the app — free to try.

Read Full Summary →

Free • No credit card required

Listen to Audio Summary Open in App

Never miss an episode of Developer Voices

Get every new episode summarized in your inbox — free, ~5 minutes to read.

No spam. Unsubscribe anytime.

What you'll learn

1 (00:00) **The Cost of Bad Observability** - Chris sets up the episode with a story about a company paying $2M for server support, contrasting it with modern distributed systems problems.
2 (03:18) **Introducing Mohammed Aboullaite** - The guest is introduced, and they establish the context of his experience at Spotify and his focus on Linux-based profiling.
3 (06:49) **The Shift to Continuous Profiling** - Chris describes his outdated view of profiling (recompiling with flags, gathering logs), and Mohammed explains how the state of the art has moved on.
4 (09:22) **The Data Volume Problem** - Chris immediately flags the massive data generation as a potential showstopper for continuous profiling.
5 (13:49) **What is eBPF?** - Mohammed explains the origin and core mechanism of eBPF, clarifying that it is far more than a packet filter.
6 (21:04) **How eBPF Enables Profiling** - The conversation moves to how eBPF specifically instruments running applications to collect profiling data.
7 (25:25) **Connecting Kernel Calls to Source Code** - Chris asks the critical question: how does a kernel-level tool know that a `malloc` came from line 27 of a Python script?

+ Full timestamped outline available in the app

Show Notes

How do you monitor distributed systems that span dozens of microservices, multiple languages, and different databases? The old approach of gathering logs from different machines and recompiling apps with profiling flags doesn't scale when you're running thousands of servers. You need a unified strategy that works everywhere, on every component, in every language—and that means tackling the problem from the kernel level up.

Mohammed Aboullaite is a backend engineer at Spotify, and he joins us to explore the latest in continuous profiling and observability using eBPF. We dive into how eBPF lets you programmatically peek into the Linux kernel without recompiling it, why companies like Google and Meta run profiling across their entire infrastructure, and how to manage the massive data volumes that continuous profiling generates. Mohammed walks through specific tools like Pyroscope, Pixie, and Parca, explains the security model of loading code into the kernel, and shares practical advice on overhead thresholds, storage strategies, and getting organizational buy-in for continuous profiling.

Whether you're debugging performance issues, optimizing for scale, or just want to see what your code is really doing in production, this episode covers everything from packet filters to cultural changes in service of getting a clear view of your software when it hits production.

---

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@DeveloperVoices/join

eBPF: https://ebpf.io/

Google-Wide Profiling Paper (2010): https://research.google.com/pubs/archive/36575.pdf

Google pprof: https://github.com/google/pprof

Continuous Profiling Tools:

Pyroscope (Grafana): https://grafana.com/oss/pyroscope/

Pixie (CNCF): https://px.dev/

Parca: https://www.parca.dev/

Datadog Continuous Profiler: https://www.datadoghq.com/product/code-profiling/

Supporting Technologies:

OpenTelemetry: https://opentelemetry.io/

Grafana: https://grafana.com/

New Relic: https://newrelic.com/

Envoy Proxy: https://www.envoyproxy.io/

Spring Cloud Sleuth: https://spring.io/projects/spring-cloud-sleuth

Mohammed Aboullaite:

LinkedIn: https://www.linkedin.com/in/aboullaite/

GitHub: https://github.com/aboullaite

Website: http://aboullaite.me

Twitter/X: https://twitter.com/laytoun

Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/