Sam Newman on Software Factories: Why AI Can't Fix Your Production Incidents (Yet)

Sam Newman, author of Building Microservices and the new Building Resilient Distributed Systems, joins Benedikt Stemmildt to talk about resilience engineering now that AI writes more of our software and runs inside more of our products.

Resilience isn't about making sure nothing goes wrong. It's about making sure as many things go right as possible. From Google's Velcro-strapped servers to a network outage caused by rabbits eating cables, Sam explains why systems, and the people who run them, have to be built for surprise. Then he applies the same lens to AI: model providers that aren't yet cloud-grade, why automated remediation with AI is still a long way off, and why your software factory is a production system in its own right.

In this episode:

• Why resilience focuses on what goes right, not only on preventing failure
• David Woods' four concepts: robustness, rebound, graceful extensibility and sustained adaptability
• Why LLM providers aren't cloud-grade yet, and how to hedge across models and vendors
• Why AI can help during an incident, but can't fix it for you (yet)
• Why you can't one-shot a large-scale system, and the modular on-ramp to software factories
• Team Topologies vs. top-down AI mandates
• Ship fast enough to care: burnout, cognitive debt and Peter Naur's theory building

Chapters:
00:00 Intro
01:17 Sam's New Book on Resilient Distributed Systems
06:10 Making Sure Things Go Right
08:33 Why Failure Is a Numbers Game
11:31 Robustness and Rebound
13:20 LLM Providers Aren't Cloud-Grade Yet
16:04 Multi-Model, Multi-Vendor
19:45 Why AI Auto-Remediation Is a Long Way Off
24:10 Agent Retries vs. Production Retries
28:24 Building With AI vs. Building AI Into Products
30:04 Rabbits, Burning SSDs and the Unexpected
32:46 Sustained Adaptability and Psychological Safety
34:10 Is a Software Factory Resilient?
40:10 Dark Factories and the Modular On-Ramp
44:27 Teams, Not Top-Down AI Mandates
49:56 Ship Fast Enough to Care
52:15 Cognitive Debt and Theory Building
55:21 Don't Go All In
57:47 Wrap-Up

Mentioned in this episode:

• Building Resilient Distributed Systems: https://www.oreilly.com/library/view/building-resilient-distributed/9781098163532/
• Four Concepts for Resilience, David Woods: https://www.sciencedirect.com/science/article/abs/pii/S0951832015000848
• Cognitive debt, Margaret-Anne Storey: https://margaretstorey.com/blog/2026/02/09/cognitive-debt/

Guest: Sam Newman · https://samnewman.io
Host: Benedikt Stemmildt, founder of hackers&wizards

Rethink Engineering is powered by re:cinq and hackers&wizards.
All episodes: https://re-cinq.com/podcast