Blocking the bots is step one. People Inc.'s Jonathan Roberts on what comes next: the companies reselling publisher content, who sets the price, and why AI is spending its money in the wrong place.
Pete Pachal talks with Jonathan Roberts, chief innovation officer at People Inc. (People, InStyle, Investopedia), about blocking every bot by default and allowing 38 through, the disguises blocked bots wear, why robots.txt is a chocolate teapot, the "picks and shovels" companies taking content for free and selling it for money, the 18 Napsters of content with no Spotify, serving AI companies Markdown instead of HTML, who gets to set the price, per crawl versus per use, why models without reference are fiction writers, why inputs now matter more than the model, TIME and Mobian's ads for bots, and why automating the chore should never mean automating the choice.
Chapters
0:00 Why the demand for information moved upstream
2:30 Jonathan Roberts, welcome back to the Media Copilot
2:36 Can smaller publishers really block bots?
4:29 Yes, and the future is hungrier for content than ever
6:47 Why robots.txt is a chocolate teapot
8:17 The CDNs are the new front line
9:48 Block everything, allow 38
10:30 Blocked bots put on disguises
12:11 Botnets, hijacked smart TVs and the FBI
13:38 What a smaller publisher can do today
17:02 Chips, power and information
19:21 18 Napsters and no Spotify
22:14 A better product for bots than HTML
25:45 Who gets to set the price?
28:40 Per use or per crawl?
32:05 Information is a supply chain, not a mine
34:54 Training, inference and fiction writers
39:41 TIME, Mobian and ads for bots
42:49 Automate the chore, not the choice
43:24 Search is humble, chatbots are arrogant
47:03 People Inc.'s LLM visibility program
48:12 Two futures for publishers
50:39 Outro
Guest
Jonathan Roberts, Chief Innovation Officer, People Inc.
People Inc.
Mentioned
Publishers are fighting the wrong battle against AI (Pete's column)
Publishers finally have AI buyers but no say in the price
What TIME learned from a month of selling ads to bots, with Mark Howard
Jonathan's first episode, April 2025
Watch on YouTube | Full post and transcript | Subscribe on Substack
Produced by The Media Copilot.
Music: "Favorite" by Alexander Nakarada, licensed under CC BY 4.0
