This episode examines LLMLingua, a method for compressing prompts five to twenty times by using a small local language model to delete low-information tokens, so that a black-box model such as GPT-3.5 can still answer well. It explains why prompts get so long (chain-of-thought demonstrations, retrieved documents, conversation history). It also explains why quantization, pruning, and learned soft-prompt compression don't work on closed APIs. The discussion ties token surprisal and perplexity to Shannon's redundancy argument and the view of language modeling as compression. It then shows why the Selective-Context baseline fails: it scores every token independently, so a phrase repeated across demonstrations can be deleted entirely. The episode walks through the paper's components, including aligning the small model to the target and a budget controller that barely trims the instruction and question but cuts the demonstrations heavily. Along the way it flags caveats: predictable tokens aren't always unimportant, scoring is left-to-right only, and the alignment step adds under a point on GSM8K.
Sources:
1. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023
http://arxiv.org/abs/2310.05736
2. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023 (EMNLP 2023)
https://scholar.google.com/scholar?q=LLMLingua%3A+Compressing+Prompts+for+Accelerated+Inference+of+Large+Language+Models
3. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression — Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023 (ACL 2024)
https://scholar.google.com/scholar?q=LongLLMLingua%3A+Accelerating+and+Enhancing+LLMs+in+Long+Context+Scenarios+via+Prompt+Compression
4. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression — Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, et al., 2024 (ACL Findings 2024)
https://scholar.google.com/scholar?q=LLMLingua-2%3A+Data+Distillation+for+Efficient+and+Faithful+Task-Agnostic+Prompt+Compression
5. Compressing Context to Enhance Inference Efficiency of Large Language Models (Selective Context) — Yucheng Li, Bo Dong, Frank Guerin, Chenghua Lin, 2023 (EMNLP 2023)
https://scholar.google.com/scholar?q=Compressing+Context+to+Enhance+Inference+Efficiency+of+Large+Language+Models+%28Selective+Context%29
6. Prediction and Entropy of Printed English — Claude E. Shannon, 1951
https://scholar.google.com/scholar?q=Prediction+and+Entropy+of+Printed+English
7. Language Models are Unsupervised Multitask Learners (GPT-2) and perplexity-based data filtering in CCNet — Radford et al. (GPT-2, 2019); Wenzek et al. (CCNet, 2020), 2019/2020
https://scholar.google.com/scholar?q=Language+Models+are+Unsupervised+Multitask+Learners+%28GPT-2%29+and+perplexity-based+data+filtering+in+CCNet
8. Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering (Selective Context) — Yucheng Li, 2023
https://scholar.google.com/scholar?q=Unlocking+Context+Constraints+of+LLMs%3A+Enhancing+Context+Efficiency+of+LLMs+with+Self-Information-Based+Content+Filtering+%28Selective+Context%29
9. Language Modeling Is Compression — Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, et al., 2023
https://scholar.google.com/scholar?q=Language+Modeling+Is+Compression
10. Learning to Compress Prompts with Gist Tokens — Jesse Mu, Xiang Lisa Li, Noah Goodman, 2023
https://scholar.google.com/scholar?q=Learning+to+Compress+Prompts+with+Gist+Tokens
11. Adapting Language Models to Compress Contexts (AutoCompressors) — Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen, 2023
https://scholar.google.com/scholar?q=Adapting+Language+Models+to+Compress+Contexts+%28AutoCompressors%29
12. In-context Autoencoder for Context Compression in a Large Language Model (ICAE) — Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, Furu Wei, 2023
https://scholar.google.com/scholar?q=In-context+Autoencoder+for+Context+Compression+in+a+Large+Language+Model+%28ICAE%29
13. Semantic Compression with Large Language Models — Henry Gilbert, Michael Sandborn, Douglas C. Schmidt, Jesse Spencer-Smith, Jules White, 2023
https://scholar.google.com/scholar?q=Semantic+Compression+with+Large+Language+Models
14. Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance — Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, Tushar Khot, 2023
https://scholar.google.com/scholar?q=Chain-of-Thought+Hub%3A+A+Continuous+Effort+to+Measure+Large+Language+Models%27+Reasoning+Performance
15. Training Verifiers to Solve Math Word Problems (GSM8K) — Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, et al., 2021
https://scholar.google.com/scholar?q=Training+Verifiers+to+Solve+Math+Word+Problems+%28GSM8K%29
16. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, et al., 2023
https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts
17. Efficient Streaming Language Models with Attention Sinks / H2O / SnapKV (KV-cache eviction line) — Xiao et al.; Zhang et al.; Li et al., 2023-2024
https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%2F+H2O+%2F+SnapKV+%28KV-cache+eviction+line%29
Interactive Visualization: LLMLingua: Compressing Prompts with Small-Model Token Pruning
