This episode explores the 2017 Swish paper and asks whether a simple self-gated activation, `x * sigmoid(x)`, can outperform ReLU without changing the surrounding network architecture. It explains why activation functions matter for gradient flow and deep optimization, focusing on Swish’s smooth, non-monotonic behavior and its ability to attenuate rather than discard negative inputs. The discussion walks through results on CIFAR, ImageNet, and machine translation, highlighting modest but real gains in deeper vision models, including roughly 0.9-point and 0.6-point improvements on ImageNet benchmarks. It also gives a critical read of the evidence, noting that Swish is not a universal win and raises practical questions around tuning, sparsity, hardware efficiency, compression, and whether its legacy matters more as part of broader gating mechanisms than as a standalone ReLU replacement.
Sources:
1. Swish: a Self-Gated Activation Function — Prajit Ramachandran, Barret Zoph, Quoc V. Le, 2017
http://arxiv.org/abs/1710.05941v1
2. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification — Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, 2015
https://scholar.google.com/scholar?q=Delving+Deep+into+Rectifiers:+Surpassing+Human-Level+Performance+on+ImageNet+Classification
3. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) — Djork-Arne Clevert, Thomas Unterthiner, Sepp Hochreiter, 2015
https://scholar.google.com/scholar?q=Fast+and+Accurate+Deep+Network+Learning+by+Exponential+Linear+Units+(ELUs)
4. Gaussian Error Linear Units (GELUs) — Dan Hendrycks, Kevin Gimpel, 2016
https://scholar.google.com/scholar?q=Gaussian+Error+Linear+Units+(GELUs)
5. Searching for Activation Functions — Prajit Ramachandran, Barret Zoph, Quoc V. Le, 2017
https://scholar.google.com/scholar?q=Searching+for+Activation+Functions
6. Language Modeling with Gated Convolutional Networks — Yann N. Dauphin, Angela Fan, Michael Auli, David Grangier, 2016
https://scholar.google.com/scholar?q=Language+Modeling+with+Gated+Convolutional+Networks
7. Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning — Stefan Elfwing, Eiji Uchibe, Kenji Doya, 2017
https://scholar.google.com/scholar?q=Sigmoid-Weighted+Linear+Units+for+Neural+Network+Function+Approximation+in+Reinforcement+Learning
8. GLU Variants Improve Transformer — Noam Shazeer, 2020
https://scholar.google.com/scholar?q=GLU+Variants+Improve+Transformer
9. Attention Is All You Need — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, 2017
https://scholar.google.com/scholar?q=Attention+Is+All+You+Need
10. ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models — Iman Mirzadeh et al., 2023
https://arxiv.org/abs/2310.04564
11. ReLU^2 Wins: Discovering Efficient Activation Functions for Sparse LLMs — Zhengyan Zhang et al., 2024
https://arxiv.org/abs/2402.03804
12. Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts — Huy Nguyen, Nhat Ho, Alessandro Rinaldo, 2024
https://arxiv.org/abs/2405.13997
13. Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free — Zihan Qiu et al., 2025
https://arxiv.org/abs/2505.06708
14. A Flexible Template for Edge Generative AI with High-Accuracy Accelerated Softmax & GELU — Andrea Belano et al., 2024
https://arxiv.org/abs/2412.06321
15. AI Post Transformers: Adam: A Method for Stochastic Optimization — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/adam-a-method-for-stochastic-optimization/
16. AI Post Transformers: PALOMA: Benchmarking Language Model Fit Across Domains — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-23-paloma-benchmarking-language-model-fit-a-360060.mp3
