This episode explores Noam Shazeer’s 2020 paper on replacing the Transformer’s standard feed-forward network with gated linear unit variants such as GLU, Bilinear, ReGLU, GEGLU, and SwiGLU. It explains why this seemingly small change matters, walking through the role of the per-token MLP in a Transformer and how multiplicative gating can change feature processing without altering the broader encoder-decoder architecture. The discussion focuses on the paper’s T5-style sequence-to-sequence setup, including span-corruption pretraining on C4, and on the key methodological choice to shrink gated-layer width so parameter count and FLOPs stay roughly matched with the baseline. Listeners would find it interesting because the episode connects a clean, tightly controlled ablation to a design idea that later had an outsized influence on modern Transformer architectures, while also highlighting the limits of what the experiment actually proves.
Sources:
1. GLU Variants Improve Transformer — Noam Shazeer, 2020
http://arxiv.org/abs/2002.05202
2. Language Modeling with Gated Convolutional Networks — Yann N. Dauphin, Angela Fan, Michael Auli, David Grangier, 2016
https://scholar.google.com/scholar?q=Language+Modeling+with+Gated+Convolutional+Networks
3. GLU Variants Improve Transformer — Noam Shazeer, 2020
https://scholar.google.com/scholar?q=GLU+Variants+Improve+Transformer
4. PaLM: Scaling Language Modeling with Pathways — Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, et al., 2022
https://scholar.google.com/scholar?q=PaLM:+Scaling+Language+Modeling+with+Pathways
5. Gemma 2: Improving Open Language Models at a Practical Size — Gemma Team, including Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, and collaborators, 2024
https://scholar.google.com/scholar?q=Gemma+2:+Improving+Open+Language+Models+at+a+Practical+Size
6. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — Colin Raffel et al., 2019
https://scholar.google.com/scholar?q=Exploring+the+Limits+of+Transfer+Learning+with+a+Unified+Text-to-Text+Transformer
7. Do Transformer Modifications Transfer Across Implementations and Applications? — Sharan Narang et al., 2021
https://scholar.google.com/scholar?q=Do+Transformer+Modifications+Transfer+Across+Implementations+and+Applications?
8. Transformer Feed-Forward Layers Are Key-Value Memories — Mor Geva, Roei Schuster, Jonathan Berant, Omer Levy, 2021
https://scholar.google.com/scholar?q=Transformer+Feed-Forward+Layers+Are+Key-Value+Memories
9. Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers — Zihan Qiu, Zeyu Huang, Youcheng Huang, Jie Fu, 2024
https://scholar.google.com/scholar?q=Empirical+Study+on+Updating+Key-Value+Memories+in+Transformer+Feed-forward+Layers
10. ReLU^2 Wins: Discovering Efficient Activation Functions for Sparse LLMs — Zhengyan Zhang et al., 2024
https://scholar.google.com/scholar?q=ReLU^2+Wins:+Discovering+Efficient+Activation+Functions+for+Sparse+LLMs
11. Spark Transformer: Reactivating Sparsity in FFN and Attention — Chong You et al., 2025
https://scholar.google.com/scholar?q=Spark+Transformer:+Reactivating+Sparsity+in+FFN+and+Attention
12. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp3
13. AI Post Transformers: RoBERTa: Robustly Optimized BERT Pretraining Approach — Hal Turing & Dr. Ada Shannon, Wed,
https://podcast.do-not-panic.com/episodes/roberta-robustly-optimized-bert-pretraining-approach/
14. AI Post Transformers: PALOMA: Benchmarking Language Model Fit Across Domains — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-23-paloma-benchmarking-language-model-fit-a-360060.mp3
15. AI Post Transformers: Unified Neural Scaling Laws Across Regimes — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-07-unified-neural-scaling-laws-across-regim-292e2d.mp3
