This episode explores RaReCache, a 2026 preprint on reusing a small model's KV cache inside a much larger model from the same family. It starts with the handoff cost in cascades and routers: when a cheap model escalates to a big one, the big model has to redo the full prefill. The discussion covers prior work on selective recomputation (token-level in CacheBlend, layer-level in DroidSpeak) and a closed-form ridge-regression map that translates KV entries between model sizes. The motivating result uses Qwen3-0.6B as the source and Qwen3-14B as the target on GSM8K. Plain cache mapping reaches 71.6% accuracy against 95.1% for a native prefill. The paper's argument is that the errors cluster in a few positions, so the large model only needs to recompute a fraction of tokens. The hosts also note how this repair differs from lossless speculative decoding. They flag that the calibration data comes from the same task's training split, which limits what the results show. The episode ends on why attention-based selection of positions fails: the 14B model puts most of its attention on a sink token and template text, and very little on the actual question.
Sources:
1. RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation — Sreetama Sarkar, Saptarshi Mitra, Sitao Huang, Souvik Kundu, Peter A. Beerel, 2026
http://arxiv.org/abs/2610.11358v1
2. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang, 2024 (arXiv); EuroSys 2025
https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion
3. DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving — Yuhan Liu, Yuyang Huang, Jiayi Yao, Shan Lu, Junchen Jiang and collaborators, 2024 (arXiv); later versions through 2026
https://scholar.google.com/scholar?q=DroidSpeak%3A+KV+Cache+Sharing+for+Cross-LLM+Communication+and+Multi-LLM+Serving
4. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM) — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, Ion Stoica, 2023 (SOSP)
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28vLLM%29
5. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance — Lingjiao Chen, Matei Zaharia, James Zou, 2023
https://scholar.google.com/scholar?q=FrugalGPT%3A+How+to+Use+Large+Language+Models+While+Reducing+Cost+and+Improving+Performance
6. H2O: Heavy-Hitter Oracle: Efficient Generative Inference of Large Language Models with Heavy Hitters — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, Beidi Chen, 2023 (NeurIPS)
https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Heavy+Hitters
7. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer, 2022 (NeurIPS)
https://scholar.google.com/scholar?q=LLM.int8%28%29%3A+8-bit+Matrix+Multiplication+for+Transformers+at+Scale
8. Palu: Compressing KV-Cache with Low-Rank Projection — Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, Kai-Chiang Wu, 2024 (arXiv)
https://scholar.google.com/scholar?q=Palu%3A+Compressing+KV-Cache+with+Low-Rank+Projection
9. Cross-model KV cache transfer in LLM families: A closed-form linear mapping for prefill reuse — Taekyung Heo, Rasoul Shafipour, Ritchie Zhao, et al., 2026
https://scholar.google.com/scholar?q=Cross-model+KV+cache+transfer+in+LLM+families%3A+A+closed-form+linear+mapping+for+prefill+reuse
10. Cache-to-Cache: Direct Semantic Communication Between Large Language Models — Tianyu Fu, Zihan Min, Hanling Zhang, et al., 2026
https://scholar.google.com/scholar?q=Cache-to-Cache%3A+Direct+Semantic+Communication+Between+Large+Language+Models
11. Latent Space Communication via K-V Cache Alignment — Lucio M. Dery, Zohar Yahav, Henry Prior, et al., 2026
https://scholar.google.com/scholar?q=Latent+Space+Communication+via+K-V+Cache+Alignment
12. Reduced-rank regression for the multivariate linear model — Alan Julian Izenman, 1975
https://scholar.google.com/scholar?q=Reduced-rank+regression+for+the+multivariate+linear+model
13. Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2024
https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%28StreamingLLM%29
14. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of LLMs; SnapKV: LLM Knows What You Are Looking for Before Generation — Zhenyu Zhang et al.; Yuhong Li et al., 2023 / 2024
https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+LLMs%3B+SnapKV%3A+LLM+Knows+What+You+Are+Looking+for+Before+Generation
15. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023
https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding
16. Palu: Compressing KV-Cache with Low-Rank Projection; Eigen Attention — Chi-Chih Chang et al.; Utkarsh Saxena et al., 2024
https://scholar.google.com/scholar?q=Palu%3A+Compressing+KV-Cache+with+Low-Rank+Projection%3B+Eigen+Attention
17. FrugalGPT / RouteLLM — Lingjiao Chen, Matei Zaharia, James Zou; Isaac Ong et al., 2023 / 2024
https://scholar.google.com/scholar?q=FrugalGPT+%2F+RouteLLM
Interactive Visualization: RaReCache: Rank-Disagreement Recomputation for Cross-Model KV Cache Reuse
