Every answer an AI gives you sounds equally confident, whether it's true or completely made up. That's not a bug. It's how the technology was built.
Dan Klein is CTO and co-founder of Scaled Cognition, and a professor of computer science at UC Berkeley. In this conversation with Liam, Dan breaks down what a language model actually is, why it was never designed to know the truth in the first place, and why today's AI systems have no "smells," the subtle warning signs humans usually rely on to tell good information from bad.
They get into why reinforcement learning from human feedback quietly trains models to tell people what they want to hear, how that can tip into outright deception, and why Dan believes reliability, not raw intelligence, is the biggest unsolved problem in AI today.
Key Topics Covered:
What a language model actually does at its core: next token prediction
Why LLMs are plausibility engines, not truth engines
The difference between a hallucination and a lie
Why AI mistakes have no warning signs the way bad translations or sketchy websites do
How RLHF can train models to be sycophantic instead of accurate
The "package delivery" thought experiment: when reward signals diverge from truth
Why bolting reliability onto LLMs after the fact doesn't work
How Scaled Cognition architects models around verified actions instead of raw text generation
Why bigger models aren't automatically better models
The difference between disruptive technology and scaled technology
Why startups, not incumbents, tend to drive technical breakthroughs
What metacognition is and why today's AI systems don't have it
Why Dan believes reliability is the next major frontier in AI
Episode Timestamps:
00:00 Intro
00:15 What a language model actually is
06:31 From well-formed sentences to general knowledge
08:27 Why LLMs are plausibility engines, not truth engines
12:06 How Perplexity approaches verifiable answers
12:40 Dan's background and Scaled Cognition's mission
15:16 The two anti-patterns companies use to control LLMs today
21:16 How Scaled Cognition architects models differently
23:28 Does every client need a custom-trained model?
29:12 Why prompting alone can't guarantee reliability
30:55 Modularity, contracts, and building reliable systems
34:40 Why trust and digital literacy matter beyond the enterprise
39:12 Code smells and why AI mistakes have no warning signs
41:14 Are AI companies incentivized to tell the truth?
42:55 How reinforcement learning actually works
44:35 The package delivery thought experiment
48:44 Why models are trained to be sycophantic
51:01 Where this incentive is mechanically baked into the model
53:43 Does responsibility fall back on humans?
58:10 Just be more reliable than a human, not perfectly true
1:02:59 The last major technique shift in AI
1:10:55 Why frontier labs keep scaling despite the risk of disruption
1:17:15 The future of hyper-specialized models vs. one broad model
1:19:47 Is there anything uniquely human AI can't replicate?
1:25:45 Wearing three hats: professor, researcher, and CTO
1:29:47 Why Dan does what he does
Connect with Dan on LinkedIn:https://www.linkedin.com/in/dan-klein/
Partner Links
Upgrade your AI toolkit: https://www.theaireport.ai/ai-executive-pass
Subscribe to our free newsletter: https://newsletter.theaireport.ai/subscribe
Join the community: https://community.theaireport.ai/checkout/the-ai-report-welcome-gift?coupon_code=WRTH
Learn more about your ad choices. Visit megaphone.fm/adchoices
