Query-Key Normalization Improves Low-Resource Transformer Translation

This episode examines Query-Key Normalization for Transformers, which swaps the attention dot product for a scaled cosine similarity. It reports an average gain of 0.928 BLEU across five low-resource translation pairs, using small models and corpora of roughly 10,000 to 214,000 sentence pairs. The discussion explains why unbounded dot products can push softmax toward near one-hot attention, and how L2-normalizing queries and keys bounds the scores. A learnable scale, initialized by a rule of thumb based on sequence length, then restores attention sharpness. It also places the method against the ScaleNorm, FixNorm and PreNorm baseline, and flags a confound: the baseline uses ScaleNorm while the QKNorm model uses LayerNorm. Listeners get a careful look at how much of the gain the change itself can claim.

Sources: 1. Query-Key Normalization for Transformers — Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, Yuxuan Chen, 2020 http://arxiv.org/abs/2010.04245 2. Transformers without Tears: Improving the Normalization of Self-Attention — Toan Q. Nguyen, Julian Salazar, 2019 https://scholar.google.com/scholar?q=Transformers+without+Tears%3A+Improving+the+Normalization+of+Self-Attention 3. Query-Key Normalization for Transformers — Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, Yuxuan Chen, 2020 https://scholar.google.com/scholar?q=Query-Key+Normalization+for+Transformers 4. Scaling Vision Transformers to 22 Billion Parameters — Mostafa Dehghani, Josip Djolonga, Basil Mustafa, et al. (Google Research), 2023 https://scholar.google.com/scholar?q=Scaling+Vision+Transformers+to+22+Billion+Parameters 5. Small-scale proxies for large-scale Transformer training instabilities — Mitchell Wortsman, Peter J. Liu, Lechao Xiao, et al., 2023 https://scholar.google.com/scholar?q=Small-scale+proxies+for+large-scale+Transformer+training+instabilities 6. Revisiting Low-Resource Neural Machine Translation: A Case Study — Rico Sennrich, Biao Zhang, 2019 https://scholar.google.com/scholar?q=Revisiting+Low-Resource+Neural+Machine+Translation%3A+A+Case+Study 7. Massively Multilingual Neural Machine Translation — Roee Aharoni, Melvin Johnson, Orhan Firat, 2019 https://scholar.google.com/scholar?q=Massively+Multilingual+Neural+Machine+Translation 8. When and Why Are Pre-trained Word Embeddings Useful for Neural Machine Translation? — Ye Qi, Devendra Singh Sachan, Matthieu Felix, Sarguna Padmanabhan, Graham Neubig, 2018 https://scholar.google.com/scholar?q=When+and+Why+Are+Pre-trained+Word+Embeddings+Useful+for+Neural+Machine+Translation%3F 9. No Language Left Behind: Scaling Human-Centered Machine Translation — NLLB Team (Marta R. Costa-jussà et al., Meta AI), 2022 https://scholar.google.com/scholar?q=No+Language+Left+Behind%3A+Scaling+Human-Centered+Machine+Translation 10. Query-Key Normalization for Transformers (follow-up usage) / Cosine-attention in Swin Transformer V2 — Ze Liu et al., 2022 https://scholar.google.com/scholar?q=Query-Key+Normalization+for+Transformers+%28follow-up+usage%29+%2F+Cosine-attention+in+Swin+Transformer+V2 11. On Layer Normalization in the Transformer Architecture — Ruibin Xiong et al., 2020 https://scholar.google.com/scholar?q=On+Layer+Normalization+in+the+Transformer+Architecture 12. Normalized Attention Without Probability Cage — Oliver Richter, Roger Wattenhofer, 2020 https://scholar.google.com/scholar?q=Normalized+Attention+Without+Probability+Cage 13. Attention Is Not All You Need / On the Expressivity Role of LayerNorm and attention entropy collapse (Zhai et al., 'Stabilizing Transformer Training by Preventing Attention Entropy Collapse') — Shuangfei Zhai et al., 2023 https://scholar.google.com/scholar?q=Attention+Is+Not+All+You+Need+%2F+On+the+Expressivity+Role+of+LayerNorm+and+attention+entropy+collapse+%28Zhai+et+al.%2C+%27Stabilizing+Transformer+Training+by+Preventing+Attention+Entropy+Collapse%27%29 14. Root Mean Square Layer Normalization — Biao Zhang, Rico Sennrich, 2019 https://scholar.google.com/scholar?q=Root+Mean+Square+Layer+Normalization 15. BPE-Dropout: Simple and Effective Subword Regularization — Ivan Provilkov, Dmitrii Emelianenko, Elena Voita, 2020 https://scholar.google.com/scholar?q=BPE-Dropout%3A+Simple+and+Effective+Subword+Regularization

Interactive Visualization: Query-Key Normalization Improves Low-Resource Transformer Translation