This episode explores MARCH, a recurrent memory architecture that extends Gated DeltaNet with periodic "state anchors" — frozen snapshots of the recurrent state paired with content-derived routing keys, allowing tokens to retrieve earlier versions of memory that would otherwise be overwritten. The discussion frames the core limitation of linear-attention and state-space models as a fixed-state bottleneck: unlike Transformers, which retain every key-value pair, recurrent models can only read the current, most-revised version of their compressed state, permanently losing access to earlier associations once they're gated or delta-updated away. MARCH addresses this through temporal expansion rather than capacity expansion, checkpointing state every C tokens and using a learned anchor token to generate a compact routing key describing each snapshot's content. The reported results are striking: a 94.8% improvement over vanilla Gated DeltaNet on Complex Needle-in-a-Haystack retrieval, plus gains of 47.7% on single-needle retrieval and 49.6% on RULER. Listeners interested in long-context efficiency, recurrent versus attention-based architectures, or the tradeoffs between memory compression and retrievability will find a clear, mechanistic explanation of why recurrent models forget and how content-routed checkpointing offers a fix without altering the underlying recurrence.
Sources:
1. MARCH: Scaling Recurrent Memory with Content-Routed State Anchors — Ming Zhang, Kaisen Yang, Shu Yu, Ermo Hua, Ning Ding, Xia Hu, Bowen Zhou, Chaochao Lu, Youbang Sun, 2026
http://arxiv.org/abs/2608.12435v1
2. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2024
https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time
3. Memory Caching: RNNs with Growing Memory — Ali Behrouz, Zeman Li, Yuan Deng, Peilin Zhong, Meisam Razaviyayn, Vahab Mirrokni, 2026
https://scholar.google.com/scholar?q=Memory+Caching%3A+RNNs+with+Growing+Memory
4. Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity — Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy, Matthijs Douze, Maria Lomeli, Ilze Amanda Auzina, Justin Carpentier, Gabriel Synnaeve, Hervé Jégou, 2026
https://scholar.google.com/scholar?q=Sparse+Delta+Memory%3A+Scaling+the+State+of+Linear+RNNs+through+Sparsity
5. MoM: Linear Sequence Modeling with Mixture-of-Memories — Jusen Du, Weigao Sun, Disen Lan, Jiaxi Hu, Yu Cheng, 2025
https://scholar.google.com/scholar?q=MoM%3A+Linear+Sequence+Modeling+with+Mixture-of-Memories
6. Scaling Linear Attention with Sparse State Expansion — Yuqi Pan, Yongqi An, Zheng Li, Yuhong Chou, Ruijie Zhu, Xiaohui Wang, Mingxuan Wang, Jinqiao Wang, Guoqi Li, 2025
https://scholar.google.com/scholar?q=Scaling+Linear+Attention+with+Sparse+State+Expansion
7. Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff — Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, James Zou, Atri Rudra, Christopher Ré, 2024
https://scholar.google.com/scholar?q=Simple+Linear+Attention+Language+Models+Balance+the+Recall-Throughput+Tradeoff
8. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han, 2024
https://scholar.google.com/scholar?q=Quest%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference
