This episode examines "Memory Caching: RNNs with Growing Memory," a February 2026 paper proposing that recurrent models save checkpoints of their own memory as a sequence unfolds. The aim is to recover the recall that fixed-size states lose, at a cost between linear and quadratic. It sets up the problem with needle-in-a-haystack tests, then traces how linear attention, Titans and Deep Linear Attention treat memory as a small network trained during the forward pass. The paper's variants are introduced: summing cached memories, gating them per token, routing to a sparse few, and merging their parameters in "Memory Soup." The discussion ties Memory Soup to model soups and linear mode connectivity. Averaging weights only helps if the cached memories share a loss basin, so the episode treats that as the claim to test. Listeners get a clear explanation of why recurrent models forget and a skeptical way to judge whether this fix works.
Sources:
1. Memory Caching: Giving Recurrent Models Growing Memory
https://arxiv.org/pdf/2602.24281v1
2. Memory Caching: RNNs with Growing Memory — Ali Behrouz, Zeman Li, Yuan Deng, Peilin Zhong, Meisam Razaviyayn, Vahab Mirrokni, 2026
https://scholar.google.com/scholar?q=Memory+Caching%3A+RNNs+with+Growing+Memory
3. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time — Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, et al., 2022
https://scholar.google.com/scholar?q=Model+soups%3A+averaging+weights+of+multiple+fine-tuned+models+improves+accuracy+without+increasing+inference+time
4. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2025
https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time
5. Averaging Weights Leads to Wider Optima and Better Generalization (Stochastic Weight Averaging) — Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson, 2018
https://scholar.google.com/scholar?q=Averaging+Weights+Leads+to+Wider+Optima+and+Better+Generalization+%28Stochastic+Weight+Averaging%29
6. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François Fleuret, 2020
https://scholar.google.com/scholar?q=Transformers+are+RNNs%3A+Fast+Autoregressive+Transformers+with+Linear+Attention
7. Linear Mode Connectivity and the Lottery Ticket Hypothesis — Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, Michael Carbin, 2020
https://scholar.google.com/scholar?q=Linear+Mode+Connectivity+and+the+Lottery+Ticket+Hypothesis
8. Git Re-Basin: Merging Models modulo Permutation Symmetries — Samuel K. Ainsworth, Jonathan Hayase, Siddhartha Srinivasa, 2023
https://scholar.google.com/scholar?q=Git+Re-Basin%3A+Merging+Models+modulo+Permutation+Symmetries
9. Averaging Weights Leads to Wider Optima and Better Generalization (SWA) — Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson, 2018
https://scholar.google.com/scholar?q=Averaging+Weights+Leads+to+Wider+Optima+and+Better+Generalization+%28SWA%29
10. Editing Models with Task Arithmetic; TIES-Merging; Model Breadcrumbs — Ilharco et al. 2023; Yadav et al. 2023; Davari & Belilovsky 2023, 2023
https://scholar.google.com/scholar?q=Editing+Models+with+Task+Arithmetic%3B+TIES-Merging%3B+Model+Breadcrumbs
11. Log-Linear Attention — Han Guo, Songlin Yang, Tarushii Goel, Eric P. Xing, Tri Dao, Yoon Kim, 2025
https://scholar.google.com/scholar?q=Log-Linear+Attention
12. Test-Time Training with Self-Supervision / Learning to (Learn at Test Time): RNNs with Expressive Hidden States — Yu Sun, Xinhao Li, Karan Dalal, et al., 2024
https://scholar.google.com/scholar?q=Test-Time+Training+with+Self-Supervision+%2F+Learning+to+%28Learn+at+Test+Time%29%3A+RNNs+with+Expressive+Hidden+States
13. Gated Delta Networks: Improving Mamba2 with Delta Rule — Songlin Yang, Jan Kautz, Ali Hatamizadeh, 2025
https://scholar.google.com/scholar?q=Gated+Delta+Networks%3A+Improving+Mamba2+with+Delta+Rule
14. Zoology: Measuring and Improving Recall in Efficient Language Models (MQAR); Based — Simran Arora et al., 2024
https://scholar.google.com/scholar?q=Zoology%3A+Measuring+and+Improving+Recall+in+Efficient+Language+Models+%28MQAR%29%3B+Based
15. Transformer-XL; Infini-attention; Compressive Transformer — Dai et al. 2019; Munkhdalai et al. 2024; Rae et al. 2020, 2019-2024
https://scholar.google.com/scholar?q=Transformer-XL%3B+Infini-attention%3B+Compressive+Transformer
16. Mixture of Block Attention (MoBA); Native Sparse Attention — Lu et al. 2025; Yuan et al. 2025, 2025
https://scholar.google.com/scholar?q=Mixture+of+Block+Attention+%28MoBA%29%3B+Native+Sparse+Attention
Interactive Visualization: Memory Caching: Giving Recurrent Models Growing Memory
