KV Cache Translation Across Heterogeneous Large Language Models

This episode examines a paper on translating the KV cache, the per-layer key and value tensors a transformer builds while reading a document, from one model family into another. The aim is for a second model to answer as if it had read the text itself, so a multi-model system reads a document once instead of once per model. The discussion explains why caches are private to each model and why prefill cost matters for time to first token in cache-augmented generation. It also surveys related work, including latent-space alignment, Cache-to-Cache projection, and Cartridges as a non-translation point of comparison. The central idea is a tension between propagation error when translating early and correction-deficit error when translating late. This is illustrated by a U-shaped loss curve on a small same-family GPT-2 setup, and the hosts caution that it motivates the design more than it proves the result generalizes. The episode then walks through the five-step inference pipeline and begins the Mixture-of-Translators idea, which replaces a single translator with several. Listeners get a clear, skeptical read on how to interpret the headline recovery numbers against a native-prefill baseline.

Sources: 1. KV Cache Translation across Heterogeneous Large Language Models — Jin-woo Lee, Minkyung Song, Junghyun Oh, Seunghoon Han, Gwangseon Jang, Soyoung Park, Sungsu Lim, 2026 http://arxiv.org/abs/2607.28979v2 2. Cache-to-Cache: Direct Semantic Communication Between Large Language Models — Tianyu Fu et al., 2025 https://scholar.google.com/scholar?q=Cache-to-Cache%3A+Direct+Semantic+Communication+Between+Large+Language+Models 3. DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving — Yuhan Liu et al., 2024 https://scholar.google.com/scholar?q=DroidSpeak%3A+KV+Cache+Sharing+for+Cross-LLM+Communication+and+Multi-LLM+Serving 4. Communicating Activations Between Language Model Agents — Vignav Ramesh, Kenneth Li, 2025 https://scholar.google.com/scholar?q=Communicating+Activations+Between+Language+Model+Agents 5. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, et al., 2024 (EuroSys 2025) https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion 6. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon et al., 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention 7. Distilling the Knowledge in a Neural Network — Geoffrey Hinton, Oriol Vinyals, Jeff Dean, 2015 https://scholar.google.com/scholar?q=Distilling+the+Knowledge+in+a+Neural+Network 8. FitNets: Hints for Thin Deep Nets — Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, et al., 2015 https://scholar.google.com/scholar?q=FitNets%3A+Hints+for+Thin+Deep+Nets 9. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes — Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, et al., 2024 https://scholar.google.com/scholar?q=On-Policy+Distillation+of+Language+Models%3A+Learning+from+Self-Generated+Mistakes 10. Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study — Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, et al., 2025 https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+General-Purpose+Long+Context+Representations+via+Self-Study 11. Cache-to-Cache (C2C): Direct semantic communication between LLMs via KV-cache fusion — Fu et al., 2026 https://scholar.google.com/scholar?q=Cache-to-Cache+%28C2C%29%3A+Direct+semantic+communication+between+LLMs+via+KV-cache+fusion 12. Latent space communication via K-V cache alignment — Dery et al., 2026 https://scholar.google.com/scholar?q=Latent+space+communication+via+K-V+cache+alignment 13. KVComm: Selective KV sharing between LLMs — Shi et al., 2026 https://scholar.google.com/scholar?q=KVComm%3A+Selective+KV+sharing+between+LLMs 14. Enabling agents to communicate entirely in latent space (Interlat) — Du et al., 2026 https://scholar.google.com/scholar?q=Enabling+agents+to+communicate+entirely+in+latent+space+%28Interlat%29 15. Model stitching: Looking for functional similarity between representations — Bansal, Nakkiran, Barak, 2021 https://scholar.google.com/scholar?q=Model+stitching%3A+Looking+for+functional+similarity+between+representations 16. A reduction of imitation learning and structured prediction to no-regret online learning (DAgger) — Ross, Gordon, Bagnell, 2011 https://scholar.google.com/scholar?q=A+reduction+of+imitation+learning+and+structured+prediction+to+no-regret+online+learning+%28DAgger%29 17. Efficient memory management for LLM serving with PagedAttention — Kwon et al., 2023 https://scholar.google.com/scholar?q=Efficient+memory+management+for+LLM+serving+with+PagedAttention

Interactive Visualization: KV Cache Translation Across Heterogeneous Large Language Models