This episode examines Raven, a sequence-modeling architecture from EPFL, Carnegie Mellon, MBZUAI, and Cartesia AI that claims near-perfect recall at sixteen times its training context length. The discussion traces the underlying tradeoff across three existing approaches — Transformers' unbounded but expensive KV cache, state-space models like Mamba that compress history into a fixed-size state at the cost of interference between tokens, and sliding-window attention's exact but position-based eviction — and frames linear attention as mathematically equivalent to the SSM case. Raven's contribution is "Routing Slot Memories": a fixed set of memory slots where a learned, content-based router (using sigmoid scoring, top-K selection, and Gumbel noise during training) decides which slots get written and decayed, recasting SSMs and sliding-window attention as special cases of one general mechanism. Listeners interested in long-context modeling will find the reframing valuable for understanding why linear models forget distant facts and how architecture choices about "where to write memory" — not just how much to compress — determine recall performance.
Sources:
1. Raven: High-Recall Sequence Modeling with Sparse Memory Routing — Arshia Afzal, Aviv Bick, Eric P. Xing, Volkan Cevher, Albert Gu, 2026
http://arxiv.org/abs/2607.25357
2. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Albert Gu, Tri Dao, 2023
https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces
3. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François Fleuret, 2020
https://scholar.google.com/scholar?q=Transformers+are+RNNs%3A+Fast+Autoregressive+Transformers+with+Linear+Attention
4. Longformer: The Long-Document Transformer — Iz Beltagy, Matthew E. Peters, Arman Cohan, 2020
https://scholar.google.com/scholar?q=Longformer%3A+The+Long-Document+Transformer
5. Neural Turing Machines — Alex Graves, Greg Wayne, Ivo Danihelka, 2014
https://scholar.google.com/scholar?q=Neural+Turing+Machines
6. Simple linear attention language models balance the recall-throughput tradeoff (Based) — Simran Arora, Sabri Eyuboglu, Michael Zhang, et al., 2024
https://scholar.google.com/scholar?q=Simple+linear+attention+language+models+balance+the+recall-throughput+tradeoff+%28Based%29
7. The Illusion of State in State-Space Models — William Merrill, Jackson Petty, Ashish Sabharwal, 2024
https://scholar.google.com/scholar?q=The+Illusion+of+State+in+State-Space+Models
8. Stuffed Mamba: Oversized States Lead to the Inability to Forget — Yingfa Chen, Xinrong Zhang, Shengding Hu, et al., 2025
https://scholar.google.com/scholar?q=Stuffed+Mamba%3A+Oversized+States+Lead+to+the+Inability+to+Forget
9. Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism — Aviv Bick, Eric Xing, Albert Gu, 2025
https://scholar.google.com/scholar?q=Understanding+the+Skill+Gap+in+Recurrent+Language+Models%3A+The+Role+of+the+Gather-and-Aggregate+Mechanism
10. MoM: Linear Sequence Modeling with Mixture-of-Memories — Jusen Du, Weigao Sun, Disen Lan, Jiaxi Hu, Yu Cheng, 2025
https://scholar.google.com/scholar?q=MoM%3A+Linear+Sequence+Modeling+with+Mixture-of-Memories
11. Scaling Linear Attention with Sparse State Expansion (SSE) — Yuqi Pan, Yongqi An, Zheng Li, et al., 2025
https://scholar.google.com/scholar?q=Scaling+Linear+Attention+with+Sparse+State+Expansion+%28SSE%29
