Rethinking LLMs: Why Diffusion Could Change How We Build AI | Dr. Subham Sahoo

Subham Sahoo is a Research Scientist and Team Lead at the Institute of Foundation Models at MBZUAI, where he leads the Diffusion LLM team. His research focuses on discrete diffusion language models and the question of whether diffusion can offer a fundamentally different, and potentially more efficient way of generating language compared with traditional autoregressive models.Prior to that, Subham completed his Ph.D. at Cornell Tech. And his recent work includes studies of how diffusion models could be scaled and evaluated against autoregressive models. His work has helped push diffusion LMs from an interesting alternative toward a much more serious research direction for language generation.00:00:00 Highlight & Introduction00:02:10 Journey into AI & First Interest in Diffusion Models00:11:37 Learning the Mathematics Behind Diffusion Models00:16:58 Research, Publishing Pressure & Finding the Right Validation00:20:31 How to Filter Noise from AI Research00:23:32 Academia vs. Industry After a PhD00:26:32 Diffusion Models vs. Autoregressive LLMs00:29:01 How Blockwise Diffusion Solves the Latency Problem00:30:57 Why Autoregressive LLMs Still Have Room for Improvement00:38:00 How Hybrid Diffusion + Autoregressive Models Work00:41:50 Can Blockwise Prediction Preserve Model Quality?00:44:03 Will Diffusion Replace Autoregressive LLMs?00:47:52 Does Diffusion Training Make LLMs Learn Gibberish?00:50:45 Comparing Training Costs & Scaling00:55:00 How Hybrid Models Scale with More Data00:59:02 Lossy vs. Lossless Speedups01:02:09 Are Hybrid Diffusion Models Production Ready?01:04:54 How to Choose a Good AI Research Direction01:08:06 Advice for Early-Career AI Researchers01:11:39 Closing ThoughtsAbout the Host:Jay is a Machine Learning Engineer III at PathAI working on improving AI for medical diagnosis and prognosis. Linkedin: https://www.linkedin.com/in/shahjay22/Twitter: https://twitter.com/jaygshah22Homepage: https://jaygshah.github.io/ for any queries.Stay tuned for upcoming webinars!***Disclaimer: The information in this video represents the views and opinions of the speaker and does not necessarily represent the views or opinions of any institution. It does not constitute an endorsement by any Institution or its affiliates of such video content.***