Episode 240 - Jason is Back

Jason Armin returns to AB Testing after a five-year hiatus, and the conversation quickly lands on the one thing that has changed everything: AI is reshaping engineering workflows at a pace that makes previous transformations look tame. Armin has written two books over the summer—Testing AI and a companion on how AI tests software—and the episode digs into the practical, psychological, and infrastructural consequences of the current moment.

The 10x Problem That Is Really a 100x Problem

Armin draws on data from Anthropic, Google, and a major CI/CD company to make a stark observation: the number of pull requests in software organizations has already increased by roughly 10x, and that is a conservative estimate. Google has estimated that the testing demand required to keep up with this volume of code change grows quadratically on top of that exponential increase in PRs. The result is a massive backlog of business value—code that exists but cannot be shipped because teams cannot trust it enough to release. Armin predicts that within 12 to 18 months, as much as 80 percent of data center compute will be consumed by testing work, not code generation. The CI/CD provider he spoke with is seeing 30 percent growth in demand, but customers are pushing back on costs—their bills are up, but their need has grown 10x, and they cannot afford the compute nor the tokens. The irony is that