Stream's AI support agent resolves 80% of their customer tickets. It was built by one engineer, working part-time, inside a 60-person product and engineering team.
In hundreds of conversations with CX leaders evaluating self-build, this is the exception. Most attempts stall between an impressive hackathon demo and a system reliable enough for production. Nick Rogers, Chief Product and Technology Officer at stream (formerly Wagestream), explains why his team is one of the few that crossed that gap.
Nick walks through the test he'd give any leader weighing build versus buy (his answer: if you need to ask, partner), the "wait for the frontier" tempo of trying, failing, and coming back when a new model release shifts what's possible, and the results of stream's recent model bake-off, including the model that surprised the team for being worse than expected.
Follow The Squawk podcast for hot takes and big ideas in CX. Perfect for a short commute and guaranteed to leave you with something new to think about, or implement, for your CX operation.
Timestamps:
00:00 - Cold open
00:41 - Intro to Nick & stream
01:55 - Nick's career path
04:00 - Why stream's self-build is the exception
05:07 - The chronological story: GPT-3.5 to Gemini 2.0 Flash
07:50 - The culture that lets stream try and fail
09:15 - Eval methodology: golden datasets and A/B testing in production
11:30 - 70% to 80% deflection, a 33% reduction in tickets
11:55 - The model bake-off and the Claude Opus surprise
13:20 - Latency versus quality tradeoffs
14:46 - "How many of your 60 engineers work on this?" - effectively one
16:18 - Why talent matters (Joel on Software's "high notes")
17:39 - The self-build vs buy test: "if you need to ask, just buy"
19:30 - The "wait for the frontier" strategy
20:58 - Risk: from wild hallucination to agentic actions
24:41 - Knowledge management as the new CX job
26:42 - Lightning round
28:51 - Recommended reading: Hackers and Painters by Paul Graham
29:48 - Wrap-up
Where to find Nick:
LinkedIn: https://www.linkedin.com/in/rogersnm/
