Compute scarcity is an engineering problem

Angelos Perivolaropoulos, a research engineer at ElevenLabs, on turning GPU scarcity into an inference-engineering problem: how to serve far more users on the same hardware, from batching to frontier architecture changes. Recorded at RAAIS 2026.

00:00 Introduction: ElevenLabs and the GPU squeeze

00:38 The question: how to scale when you can't add capacity

01:11 About Angelos: Scribe, speech-to-text and text-to-speech

01:56 GPU scarcity meets exponential demand

02:44 What a token actually costs: compute vs memory bandwidth

03:38 Prefill, decode and the KV cache

05:53 Batching and continuous batching (1 → 15 users/GPU)

08:37 FP8 quantization and quantize-aware training (→ 20)

11:29 Speculative decoding and multi-token prediction (→ 28)

15:13 Compressing the KV cache: TurboQuant and distillation (→ 70)

17:27 Frontier architectures: MLA, linear attention, state-space (→ 140)

20:39 Trade-offs: nothing is free

22:03 Q&A: papers vs production, token subsidies, TTS evals