This episode examines "Speculative Prefill," an ICML 2025 method that cuts time to first token by having a small model (Llama-3.1-8B) estimate which prompt tokens matter, so a large model (Llama-405B) prefills only those. It explains why prefill dominates in real traffic, where prompts are often about ten times longer than outputs. It also explains why dropping tokens differs from sparse-attention approaches like MInference, since it also removes MLP and tensor-parallel communication costs. The discussion places the work among related approaches (SnapKV, LazyLLM, GemFilter, SwiftKV, LLMLingua) as a training-free method that doesn't rely on the big model to pick tokens. It also spells out the two bets behind it: that token importance transfers between models, and that the big model tolerates a gappy prompt at a given keep rate. The episode cautions that the headline 7.66x TTFT and 7x QPS figures come from different experiments and best-case conditions. It also notes that the importance score is an attention heuristic rather than a formal information-theoretic measure.
Sources:
1. Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation — Jingyu Liu, Beidi Chen, Ce Zhang, 2025
http://arxiv.org/abs/2502.02789
2. SnapKV: LLM Knows What You are Looking for Before Generation — Yuhong Li, Yingbing Huang, Bowen Yang, et al., 2024
https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+are+Looking+for+Before+Generation
3. LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference — Qichen Fu et al., 2024
https://scholar.google.com/scholar?q=LazyLLM%3A+Dynamic+Token+Pruning+for+Efficient+Long+Context+LLM+Inference
4. Not All Heads Matter / Quest, and RetrievalAttention-style dynamic sparse attention — Jiaming Tang et al. (Quest); Di Liu et al. (RetrievalAttention), 2024
https://scholar.google.com/scholar?q=Not+All+Heads+Matter+%2F+Quest%2C+and+RetrievalAttention-style+dynamic+sparse+attention
5. GemFilter: Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction — Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, Shafiq Joty, 2024
https://scholar.google.com/scholar?q=GemFilter%3A+Discovering+the+Gems+in+Early+Layers%3A+Accelerating+Long-Context+LLMs+with+1000x+Input+Token+Reduction
6. SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation — Aurick Qiao et al. (Snowflake), 2024
https://scholar.google.com/scholar?q=SwiftKV%3A+Fast+Prefill-Optimized+Inference+with+Knowledge-Preserving+Model+Transformation
7. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023
https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding
8. LLMLingua-2 / LongLLMLingua: Question-aware prompt compression — Huiqiang Jiang, Qianhui Wu, et al., 2023-2024
https://scholar.google.com/scholar?q=LLMLingua-2+%2F+LongLLMLingua%3A+Question-aware+prompt+compression
9. Sarathi-Serve / Chunked prefill and Mooncake / prefill-decode disaggregation — Agrawal et al. (Sarathi-Serve); Qin et al. (Mooncake), 2024
https://scholar.google.com/scholar?q=Sarathi-Serve+%2F+Chunked+prefill+and+Mooncake+%2F+prefill-decode+disaggregation
10. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu et al., 2023
https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts
11. Marconi / prefix-caching for hybrid LLMs and LMCache/CacheBlend — Pan et al. (Marconi); Yao et al. (CacheBlend), 2024-2025
https://scholar.google.com/scholar?q=Marconi+%2F+prefix-caching+for+hybrid+LLMs+and+LMCache%2FCacheBlend
Interactive Visualization: Speculative Prefill: Training-Free Token Importance for Faster TTFT
