This episode examines BABILong, a NeurIPS 2024 benchmark that asks how much of a language model's advertised 128K-plus context window it can use for reasoning. It argues that the popular needle-in-a-haystack test mostly measures lookup, which strong models pass almost perfectly. BABILong hides the sentences of the synthetic bAbI reasoning tasks (chaining facts, counting, temporal ordering) inside long passages of PG19 book text. Answering then requires finding and combining several scattered facts, and the benchmark can be generated at any length, up to millions of tokens, without risk of training-set contamination. The discussion defines "effective context size" as the longest input at which accuracy stays above 85%, and it flags that this threshold is a judgment call. It also sets up comparisons between long-context LLMs, retrieval-augmented generation, and small recurrent-memory and state-space models. The hosts note that some of the winning models are trained on the task and others are not, so listeners get both the benchmark's design and a caution about reading its results.
Sources:
1. BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack — Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, Mikhail Burtsev, 2024
http://arxiv.org/abs/2406.10149
2. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks (bAbI) — Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin, Tomas Mikolov, 2015 (ICLR 2016)
https://scholar.google.com/scholar?q=Towards+AI-Complete+Question+Answering%3A+A+Set+of+Prerequisite+Toy+Tasks+%28bAbI%29
3. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023 (TACL 2024)
https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts
4. RULER: What's the Real Context Size of Your Long-Context Language Models? — Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg (NVIDIA), 2024 (COLM 2024)
https://scholar.google.com/scholar?q=RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F
5. NoLiMa: Long-Context Evaluation Beyond Literal Matching — Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, Hinrich Schütze (Adobe Research / LMU), 2025
https://scholar.google.com/scholar?q=NoLiMa%3A+Long-Context+Evaluation+Beyond+Literal+Matching
6. Recurrent Memory Transformer — Aydar Bulatov, Yuri Kuratov, Mikhail Burtsev, 2022 (NeurIPS 2022)
https://scholar.google.com/scholar?q=Recurrent+Memory+Transformer
7. Scaling Transformer to 1M tokens and beyond with RMT — Aydar Bulatov, Yuri Kuratov, Yermek Kapushev, Mikhail S. Burtsev, 2023
https://scholar.google.com/scholar?q=Scaling+Transformer+to+1M+tokens+and+beyond+with+RMT
8. Associative Recurrent Memory Transformer (ARMT) — Ivan Rodkin, Yuri Kuratov, Aydar Bulatov, Mikhail Burtsev, 2024
https://scholar.google.com/scholar?q=Associative+Recurrent+Memory+Transformer+%28ARMT%29
9. Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention — Tsendsuren Munkhdalai, Manaal Faruqui, Siddharth Gopal (Google), 2024
https://scholar.google.com/scholar?q=Leave+No+Context+Behind%3A+Efficient+Infinite+Context+Transformers+with+Infini-attention
10. Associative Recurrent Memory Transformer — Rodkin, Kuratov, Bulatov, Burtsev, 2024
https://scholar.google.com/scholar?q=Associative+Recurrent+Memory+Transformer
11. Scaling Transformer to 1M tokens and beyond with RMT / Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory — Bulatov, Kuratov, Kapushev, Burtsev, 2023/2024
https://scholar.google.com/scholar?q=Scaling+Transformer+to+1M+tokens+and+beyond+with+RMT+%2F+Beyond+Attention%3A+Breaking+the+Limits+of+Transformer+Context+Length+with+Recurrent+Memory
12. Gated Delta Networks: Improving Mamba2 with Delta Rule — Yang, Kautz, Hatamizadeh, 2024
https://scholar.google.com/scholar?q=Gated+Delta+Networks%3A+Improving+Mamba2+with+Delta+Rule
13. Titans: Learning to Memorize at Test Time — Behrouz, Zhong, Mirrokni, 2024/2025
https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time
14. Transformer-XL and Compressive Transformer (PG19) — Dai et al. 2019; Rae et al. 2020, 2019/2020
https://scholar.google.com/scholar?q=Transformer-XL+and+Compressive+Transformer+%28PG19%29
15. Infini-attention (Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention) — Munkhdalai, Faruqui, Gopal, 2024
https://scholar.google.com/scholar?q=Infini-attention+%28Leave+No+Context+Behind%3A+Efficient+Infinite+Context+Transformers+with+Infini-attention%29
16. Counting-Stars and InfiniteBench — Song et al.; Zhang et al., 2024
https://scholar.google.com/scholar?q=Counting-Stars+and+InfiniteBench
Interactive Visualization: BABILong: Testing LLM Long-Context Reasoning in a Haystack
