This episode examines "Efficient Streaming Language Models with Attention Sinks" (ICLR 2024), which asks how a model trained on a fixed attention window can run over an endless input stream at constant memory and constant per-token cost, with no fine-tuning. It explains why plain window attention collapses once the first tokens are evicted from the KV cache (perplexity in the thousands), and why recomputing every window works but is slow. The key argument is that softmax forces attention to land somewhere, so models learn to park surplus attention on the initial tokens, which act as "sinks". On Llama-2-13B, keeping just the first four tokens plus a rolling window brings perplexity from about 5,158 down to 5.40, close to the 5.43 of full recomputation. Replacing those four tokens with newlines still works, which shows that absolute position matters and content doesn't. The discussion also stresses that the method delivers stable generation, not long-term memory, and it ends on how positions are assigned by cache slot rather than by original text position.
Sources:
1. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023
http://arxiv.org/abs/2309.17453
2. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023 (ICLR 2024)
https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks
3. Massive Activations in Large Language Models — Mingjie Sun, Xinlei Chen, J. Zico Kolter, Zhuang Liu, 2024
https://scholar.google.com/scholar?q=Massive+Activations+in+Large+Language+Models
4. Vision Transformers Need Registers — Timothée Darcet, Maxime Oquab, Julien Mairal, Piotr Bojanowski, 2023
https://scholar.google.com/scholar?q=Vision+Transformers+Need+Registers
5. Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing — Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort, 2023
https://scholar.google.com/scholar?q=Quantizable+Transformers%3A+Removing+Outliers+by+Helping+Attention+Heads+Do+Nothing
6. H2O: Heavy-Hitter Oracle: Efficient Generative Inference of Large Language Models with Heavy-Hitter Oracle for KV Cache — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Zhangyang Wang, Beidi Chen, et al., 2023
https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Heavy-Hitter+Oracle+for+KV+Cache
7. Longformer: The Long-Document Transformer — Iz Beltagy, Matthew E. Peters, Arman Cohan, 2020
https://scholar.google.com/scholar?q=Longformer%3A+The+Long-Document+Transformer
8. Efficiently Scaling Transformer Inference — Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, et al., 2022
https://scholar.google.com/scholar?q=Efficiently+Scaling+Transformer+Inference
9. Efficient Streaming Language Models with Attention Sinks (follow-up analyses): When Attention Sink Emerges in Language Models: An Empirical View — Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, Min Lin, 2024
https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%28follow-up+analyses%29%3A+When+Attention+Sink+Emerges+in+Language+Models%3A+An+Empirical+View
10. Attention is Off by One (Softmax1) — Evan Miller, 2023
https://scholar.google.com/scholar?q=Attention+is+Off+by+One+%28Softmax1%29
11. LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models — Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, Sinong Wang, 2023
https://scholar.google.com/scholar?q=LM-Infinite%3A+Simple+On-the-Fly+Length+Generalization+for+Large+Language+Models
12. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, Beidi Chen, 2023
https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models
13. SnapKV: LLM Knows What You are Looking for Before Generation — Yuhong Li, Yingbing Huang, Bowen Yang, et al., 2024
https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+are+Looking+for+Before+Generation
14. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023
https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts
15. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
16. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Albert Gu, Tri Dao, 2023
https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces
17. Efficient Transformers: A Survey / Longformer — Iz Beltagy, Matthew E. Peters, Arman Cohan, 2020
https://scholar.google.com/scholar?q=Efficient+Transformers%3A+A+Survey+%2F+Longformer
Interactive Visualization: Attention Sinks Enable Streaming Language Models Over Endless Input
