This episode examines "In-Context Autoencoder for Context Compression in a Large Language Model" (ICLR 2024), which uses a LoRA-adapted copy of an LLM to squeeze a context into 128 learned memory slots that an unmodified, frozen decoder reads in place of the original tokens, for roughly 4x compression. It covers the two-stage training: self-supervised pretraining on the Pile with autoencoding and language-modeling objectives, then instruction tuning on a GPT-4-generated prompt-and-answer dataset. The discussion contrasts this design with Gist tokens and AutoCompressors, and with architectural fixes for long context such as sparse attention and state-space models. It also draws on the "Lost in the Middle" findings as motivation. The hosts question whether the paper's evidence supports its "long context" framing, since compressing a single passage is a different claim from handling truly long inputs. They also flag that the synthetic training data and the GPT-4 judge deserve scrutiny. Listeners get a clear explanation of how soft-prompt-style memory slots work and a critical lens on what the results do and don't show.
Sources:
1. In-context Autoencoder for Context Compression in a Large Language Model — Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, Furu Wei, 2023
http://arxiv.org/abs/2307.06945
2. Learning to Compress Prompts with Gist Tokens — Jesse Mu, Xiang Lisa Li, Noah Goodman, 2023
https://scholar.google.com/scholar?q=Learning+to+Compress+Prompts+with+Gist+Tokens
3. Adapting Language Models to Compress Contexts — Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen, 2023
https://scholar.google.com/scholar?q=Adapting+Language+Models+to+Compress+Contexts
4. Compressive Transformers for Long-Range Sequence Modelling — Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Timothy P. Lillicrap, 2019
https://scholar.google.com/scholar?q=Compressive+Transformers+for+Long-Range+Sequence+Modelling
5. Perceiver: General Perception with Iterative Attention — Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, Joao Carreira, 2021
https://scholar.google.com/scholar?q=Perceiver%3A+General+Perception+with+Iterative+Attention
6. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023
https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts
7. Adapting Language Models to Compress Contexts (AutoCompressors) — Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen, 2023
https://scholar.google.com/scholar?q=Adapting+Language+Models+to+Compress+Contexts+%28AutoCompressors%29
8. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023
https://scholar.google.com/scholar?q=LLMLingua%3A+Compressing+Prompts+for+Accelerated+Inference+of+Large+Language+Models
9. Language Modeling Is Compression — Grégoire Delétang et al., 2023
https://scholar.google.com/scholar?q=Language+Modeling+Is+Compression
10. Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023
https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%28StreamingLLM%29
11. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of LLMs — Zhenyu Zhang et al., 2023
https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+LLMs
12. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache — Zirui Liu et al., 2024
https://scholar.google.com/scholar?q=KIVI%3A+A+Tuning-Free+Asymmetric+2bit+Quantization+for+KV+Cache
13. Efficient Memory Management for LLM Serving with PagedAttention (vLLM) — Woosuk Kwon et al., 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+LLM+Serving+with+PagedAttention+%28vLLM%29
14. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning — Tri Dao, 2023
https://scholar.google.com/scholar?q=FlashAttention-2%3A+Faster+Attention+with+Better+Parallelism+and+Work+Partitioning
15. Learning to Compress Contexts / xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token — Xin Cheng et al., 2024
https://scholar.google.com/scholar?q=Learning+to+Compress+Contexts+%2F+xRAG%3A+Extreme+Context+Compression+for+Retrieval-augmented+Generation+with+One+Token
16. In-Context Former / 500xCompressor: Generalized Prompt Compression — Zongqian Li, Yixuan Su, Nigel Collier, 2024
https://scholar.google.com/scholar?q=In-Context+Former+%2F+500xCompressor%3A+Generalized+Prompt+Compression
17. Perceiver / Flamingo (Perceiver Resampler) and BLIP-2 (Q-Former) — Andrew Jaegle et al.; Jean-Baptiste Alayrac et al.; Junnan Li et al., 2021-2023
https://scholar.google.com/scholar?q=Perceiver+%2F+Flamingo+%28Perceiver+Resampler%29+and+BLIP-2+%28Q-Former%29
18. Prompt Cache / Cartridges-style KV compression (Learning to compress context by training a compact KV cache) — In Gim et al.; Sabri Eyuboglu et al., 2024-2025
https://scholar.google.com/scholar?q=Prompt+Cache+%2F+Cartridges-style+KV+compression+%28Learning+to+compress+context+by+training+a+compact+KV+cache%29
Interactive Visualization: In-Context Autoencoder: Compressing Long Contexts into Memory Slots
