Text Diffusion Models with Brendan O'Donoghue (Google DeepMind)

Brendan O'Donoghue, research director at Google DeepMind, makes the case for text diffusion as a real alternative to autoregressive generation. He walks through how discrete diffusion works, why diffusion samples are far more diverse and what that unlocks for RL, where the Gemma diffusion model actually stands against frontier models, and why the whole training and serving stack being hyper-optimized for autoregression is the main thing holding the approach back. The conversation also covers hardware trends favoring flops over bandwidth, AGI timelines and real-world bottlenecks, and why he thinks RL is still underhyped.

Key topics
- Discrete diffusion for text vs autoregressive generation
- Why diffusion samples are more diverse, and what that unlocks for RL
- Where diffusion already wins: latency, on-device, robotics
- Why serving cost, not quality, is the real blocker
- RL as the most underhyped area in AI

Timeline
00:00 Introduction
00:50 What diffusion models are and how text diffusion works
04:40 Why Brendan bet on text diffusion in 2023
07:15 Diversity, creativity, and why it helps RL
11:00 The best diffusion LLM today and the gap to frontier models
14:25 Latency, serving cost, and why it needs more chips
17:14 Where diffusion already wins: on-device, robotics, battery
20:14 One model, two modes: diffusion for thinking, AR for answering
22:24 Samplers and the stuttering problem
26:27 Theory, BERT, and why now is a good time to work on this
31:48 Pipelines built for autoregression, and continuous diffusion
35:35 Hardware: flops vs bandwidth
39:49 AGI timelines and real-world bottlenecks
50:15 Is AI engineering or science?
54:14 Most overhyped and most underhyped ideas
58:35 RL on diffusion, value functions, and exploration
1:07:30 Go download the model and break it

Music

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.