KPMG pulled a report this year after its team accepted hallucinated information without question. Latham & Watkins submitted a court filing built on fabricated legal citations an AI invented, delivered with total confidence and perfect formatting. McDonald's AI ordering chatbot became a public failure for the same underlying reason: the system looked like it was working. Dr. Moritz Sudhof, CEO of Bigspin AI, analyzed 100,000 real conversations between users and live AI systems with Stanford NLP Group's Chris Potts and found why: 79% of AI failures are invisible to standard monitoring because they are behavioral failures, not technical ones.
Sudhof built this research on hard-won experience. As VP of AI at BetterUp, he shipped an AI coach that beat ChatGPT on every expert coaching benchmark and still lost users — until his team changed nothing but how the AI introduced itself, and outcomes doubled. That gap between what an eval measures and what actually happens in the room with a user became his research question, and then his company.
In this episode:
- The Confidence Trap: AI states something false, dressed in precise numbers and total confidence.
- Silent Mismatch: when the AI can't finish a task, it quietly answers a different question instead.
- The Drift and the Death Spiral: how a long conversation loses the plot and burns the user's patience.
- Models are post-trained to answer, not clarify — a default that gets worse as models get more capable.
- The Paradox of AI Fluency: the users who push back hardest also hit the most failures, and succeed most.
- Evals catch what you already know to test for. Reading real transcripts catches what you don't.
"The behavioral layer is where most of the damage is actually happening." — Moritz Sudhof
"The more specific and helpful it often gets, the more fake it often is." — Moritz Sudhof, on the Confidence Trap
About Moritz: Shipped conversational AI to hundreds of organizations as VP of AI at BetterUp. Lived the problem of knowing something's wrong but not what to fix.
Guest resources:
Key research:
We built productimpactpod.com to be your AI product insights and strategic playbook hub. Check it out.
Thank you for listening to the Product Impact Podcast — if you have feedback, guest recommendations, or want to chat — contact us.
Hosted by:
Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com
This episode was brought to you by:
PH1 (https://ph1.ca) — a strategy & research consultancy specialized in delivering evidence about the highest value use cases and customer profiles. AI Value Acceleration
(https://aivalueacceleration.com) — The consultancy specialising in enterprise value creation. Make sure that your spending doesn't go to waste. Find out exactly where the value creation of adopting AI products stalls.
