In Elixir Wizards S15E04, Charles Suggs and Emma Whamond are joined by Somtochi Onyekwere, a software engineer at Fly.io and contributor to the Corrosion distributed database project, to talk about distributed systems, infrastructure resilience, and the growing fragility of centralized cloud platforms.
We discuss what recent outages across major providers reveal about modern infrastructure and why more teams are starting to rethink assumptions around reliability, failover, and system design. Somtochi explains how Fly.io approaches geographic distribution, eventual consistency, and replication across nodes, along with the trade-offs that come with building systems this way.
The conversation explores CRDTs (Conflict-free Replicated Data Types), consensus, split-brain prevention, and what actually happens when distributed systems fail in production. We also talk about testing strategies, rollback planning, property-based testing tools, and how teams can reduce blast radius when things inevitably go wrong.
Along the way, we discuss AI infrastructure, sandboxing AI agents, and how newer workloads may add pressure to already centralized systems. The episode closes with practical advice for developers who want to build more resilient applications without over-complicating their architecture.
Topics Discussed in this Episode:
Corrosion and distributed database replication
Centralized cloud fragility and recent outage patterns
Distributed systems versus traditional cloud architectures
Multi-region deployment strategies for Phoenix applications
CRDTs and conflict resolution in distributed systems
Eventual consistency versus strict consistency tradeoffs
Consensus, leader election, and split-brain prevention
Testing failover and recovery scenarios
Property-based testing and Antithesis
Rollback planning for database schema migrations
Reducing blast radius through system isolation
Health checks and blue-green deployment strategies
Fly Proxy request routing and replay behavior
Cross-region synchronization and replication challenges
Single points of failure inside “redundant” systems
Backup restoration testing and disaster recovery planning
Network partitions and failure handling in production
Infrastructure monitoring and operational visibility
AI infrastructure workloads and operational strain
Sandboxing and securing AI agents
Sprites and AI workflows at Fly.io
Latency improvements from geographic distribution
Distributed systems tradeoffs in real-world environments
Transitive dependency failures across cloud providers
Practical resilience strategies for modern engineering teams
Podden och tillhörande omslagsbild på den här sidan tillhör
SmartLogic LLC. Innehållet i podden är skapat av SmartLogic LLC och inte av,
eller tillsammans med, Poddtoppen.
