Vik Sekar and Austin Lyons tackle the biggest bottleneck in inference: moving data. They break down the three tiers of datacenter networking — scale up, scale out, scale across — and the core engineering trade-off at each layer: copper vs. optics. Topics include Nvidia's extreme measures to keep scale-up fabric electrical (a 78-layer mid-plane PCB), why Co-Packaged Optics is the "holy grail" everyone wants and no one can ship, and the serviceability problem standing in its way.
Key Takeaways:
A single 72-GPU scale-up rack needs 5,000+ cables spanning ~2 km — at that density, the power and cost of every connection becomes a system-level constraint, not an implementation detail.
The scale-up rule is "copper when you can, optics when you must": each pluggable optical module adds ~30W, and with thousands of links in the fabric, that penalty compounds fast.
Nvidia is pushing copper past its usual limits with a 78-layer mid-plane PCB — 3x+ the layer count of a typical complex board — specifically to avoid paying the optics power tax in its scale-up fabric.
Scale-up isn't just a training problem. Frontier MoE models need 72-GPU domains to hit memory bandwidth targets, which pulls high-performance interconnect into the inference conversation.
Scale-up has the highest connection density of the three tiers, making it the largest TAM and the sharpest three-way fight between NVLink, UA-Link, and Ethernet.
Co-Packaged Optics could cut interconnect power by two-thirds — but a single failed laser could brick an entire multi-thousand-dollar GPU package, and that serviceability risk is what's keeping it on the roadmap instead of in racks.
Chapters: 0:00 The Biggest Problem in Computing 7:14 The Three Tiers of Networking 13:49 Scale Up: Copper vs. Optics 17:59 Front-End vs. Back-End Networks 22:12 The Physical Scale of Cabling 28:46 Nvidia's 78-Layer Mid-Plane 33:16 How Optical Transceivers Work 39:16 The Power Penalty of Pluggables 42:09 The Business of Speed Transitions 44:40 The Promise and Peril of CPO 51:45 The Holy Grail of Networking
Podden och tillhörande omslagsbild på den här sidan tillhör
Vikram Sekar and Austin Lyons. Innehållet i podden är skapat av Vikram Sekar and Austin Lyons och inte av,
eller tillsammans med, Poddtoppen.
