Surya Ganguli: The Physics of Intelligence

Surya Ganguli is a professor at Stanford and VP at General Catalyst, working at the intersection of physics, neuroscience, and AI. He started in string theory, moved to theoretical neuroscience, and now uses tools from statistical physics to understand both brains and neural networks.

We talk about why deep learning theory is finally catching up to practice,  including his group's recent work explaining neural scaling laws, and why smarter data selection could beat them entirely. He also tells the origin story of diffusion models, which were invented in his lab as an attempt to violate the second law of thermodynamics.

The second half turns to the brain: what happens to a mouse's sense of self on ketamine, how stimulating a handful of neurons can induce hallucinations, and a method his lab developed to get a neuron deep in a monkey's brain to describe, in English, what makes it fire.

We close on where he thinks AI is going wrong: models train on ten trillion tokens while humans hear a hundred million words, because we don't teach children with gradients; we tell them the algorithm.

key topics

• Connections between physics, neuroscience, and AI
• Emergent properties in complex systems
• Scaling laws in language models
• Data efficiency and pruning in AI
• Neuroscience insights into consciousness and self
• The future of AI and brain modeling

• Chapters

  00:00 Introduction to Surya Ganguli

  00:57 Surya's Background: From String Theory to Neuroscience

  02:22 Emergent Properties in Physics, Neuroscience, and AI

  03:16 Energy Landscapes and Loss Landscapes in High Dimensions

  04:07 Why Local Minima Don't Exist in High-Dimensional AI

  05:22 Gradient-Based vs. Gradient-Free Learning Methods

  08:21 AI in Mathematics and Drug Discovery: Opportunities and Challenges

  13:48 Scaling Laws and Data Efficiency in Language Models

  18:10 Properties of Data that Affect Scaling Laws

  22:04 Constructing Non-Redundant Data Sets for Better Learning

  24:32 Theory vs. Empirical Results in AI Research

  32:19 Fundamental Components of Deep Learning: Are They Changing?

  34:31 Future Paradigms in AI Beyond Current Models

  37:22 Teaching AI and Humans: Paradigm Shifts in Learning

  41:37 Consciousness, Self, and the Brain: Surya's Perspectives

  49:49 Neuroscience and AI: Understanding the Brain and Consciousness

  01:02:03 Understanding the Brain: Challenges and Opportunities

  01:09:21 Brain-Computer Interfaces and AI in Neuroscience

• Music

  • "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.