High Bandwidth Flash Is Coming

I have to be real with you. Before traveling to the Bay last month, I never heard of the words “decode”, “prefill”, or “KV Cache”. Then I go to Hot Chips, and I took a full blast in the face with these terms. And now, just like 11:11 (make a wish), I see it everywhere. It’s Baader-Meinhof to the max.

But Prefill, Decode and KV Cache relate to major technical challenges associated with serving these ever-growing models. The memory makers - flush off the HBM bonanza - have brought forth a new thing with a slick name to meet these challenges: High Bandwidth Flash or HBF. HBF is cool but ... what is it going to be used for? In today’s video, a potential challenger and/or companion to HBM: High Bandwidth Flash.