This episode examines LoCoMo, a benchmark for testing whether LLM agents remember across very long conversations. The conversations average about 9,200 tokens, 19 sessions and 305 turns, roughly nine times longer than earlier datasets such as Multi-Session Chat. The episode covers how the conversations are generated: two LLM agents get personas and a temporal event graph of up to 25 causally linked life events. Human annotators then edited about 15 percent of turns, which raises questions about how much of the data was verified. It compares three ways a model can handle history: truncated context, long-context models, and retrieval-augmented generation. It also walks through the question-answering categories (single-hop, multi-hop, temporal, open-domain, adversarial) and F1 scoring. The paper's headline finding is that every approach falls well short of human performance. The episode also previews how later memory products such as MemGPT, Mem0, and Zep use the benchmark, and the leaderboard disputes that followed.
Sources:
1. Evaluating Very Long-Term Conversational Memory of LLM Agents — Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, Yuwei Fang, 2024
http://arxiv.org/abs/2402.17753
2. Evaluating Very Long-Term Conversational Memory of LLM Agents (LoCoMo) — Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, Yuwei Fang, 2024 (ACL 2024)
https://scholar.google.com/scholar?q=Evaluating+Very+Long-Term+Conversational+Memory+of+LLM+Agents+%28LoCoMo%29
3. Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models — Qingyue Wang, Liang Ding, Yanan Cao, Zhiliang Tian, Shi Wang, Dacheng Tao, Li Guo, 2023
https://scholar.google.com/scholar?q=Recursively+Summarizing+Enables+Long-Term+Dialogue+Memory+in+Large+Language+Models
4. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez, 2023
https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems
5. Generative Agents: Interactive Simulacra of Human Behavior — Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, Michael S. Bernstein, 2023
https://scholar.google.com/scholar?q=Generative+Agents%3A+Interactive+Simulacra+of+Human+Behavior
6. Beyond Goldfish Memory: Long-Term Open-Domain Conversation (Multi-Session Chat) — Jing Xu, Arthur Szlam, Jason Weston, 2022 (ACL 2022)
https://scholar.google.com/scholar?q=Beyond+Goldfish+Memory%3A+Long-Term+Open-Domain+Conversation+%28Multi-Session+Chat%29
7. MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation — Jiazhan Feng, Qingfeng Sun, Can Xu, Pu Zhao, Yaming Yang, Chongyang Tao, Dongyan Zhao, Qingwei Lin, 2023 (ACL 2023)
https://scholar.google.com/scholar?q=MMDialog%3A+A+Large-scale+Multi-turn+Dialogue+Dataset+Towards+Multi-modal+Open-domain+Conversation
8. PhotoChat: A Human-Human Dialogue Dataset with Photo Sharing Behavior for Joint Image-Text Modeling — Xiaoxue Zang, Lijuan Liu, Maria Wang, Yang Song, Hao Zhang, Jindong Chen, 2021 (ACL 2021)
https://scholar.google.com/scholar?q=PhotoChat%3A+A+Human-Human+Dialogue+Dataset+with+Photo+Sharing+Behavior+for+Joint+Image-Text+Modeling
9. Visual Instruction Tuning (LLaVA) — Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee, 2023 (NeurIPS 2023)
https://scholar.google.com/scholar?q=Visual+Instruction+Tuning+%28LLaVA%29
10. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models — Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi, 2023 (ICML 2023)
https://scholar.google.com/scholar?q=BLIP-2%3A+Bootstrapping+Language-Image+Pre-training+with+Frozen+Image+Encoders+and+Large+Language+Models
11. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory — Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, Deshraj Yadav, 2025
https://scholar.google.com/scholar?q=Mem0%3A+Building+Production-Ready+AI+Agents+with+Scalable+Long-Term+Memory
12. Zep: A Temporal Knowledge Graph Architecture for Agent Memory — Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, Daniel Chalef, 2025
https://scholar.google.com/scholar?q=Zep%3A+A+Temporal+Knowledge+Graph+Architecture+for+Agent+Memory
13. A-Mem: Agentic Memory for LLM Agents — Wujiang Xu et al., 2025
https://scholar.google.com/scholar?q=A-Mem%3A+Agentic+Memory+for+LLM+Agents
14. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory — Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, Dong Yu, 2024
https://scholar.google.com/scholar?q=LongMemEval%3A+Benchmarking+Chat+Assistants+on+Long-Term+Interactive+Memory
15. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2024
https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts
16. Beyond Goldfish Memory: Long-Term Open-Domain Conversation (MSC) — Jing Xu, Arthur Szlam, Jason Weston, 2022
https://scholar.google.com/scholar?q=Beyond+Goldfish+Memory%3A+Long-Term+Open-Domain+Conversation+%28MSC%29
17. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2024
https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time
18. Needle-in-a-haystack and RULER: What's the Real Context Size of Your Long-Context Language Models? — Cheng-Ping Hsieh et al., 2024
https://scholar.google.com/scholar?q=Needle-in-a-haystack+and+RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F
Interactive Visualization: LoCoMo Benchmark: Do LLM Agents Remember Very Long Conversations?
