Polaris is a production of Synozur – the transformation company. Synozur reimagines business for our clients, navigating the complexities of transformation and strategy with ease.
In this episode, Dr. Craig Kaplan shares his insights on the evolution of AI, safety considerations, and the importance of democratic values in developing superintelligence. Discover how AI systems can be designed for safety and alignment with human values.
Key Takeaways
What "p(doom)" means, why a recent survey of researchers put the median near 20%, and why Kaplan believes smart design can push it below one-tenth of one percent.
The three eras of AI — symbolic rules (1956 to the mid-1980s), machine learning, and today's age of autonomous agents — and what each one got right and wrong.
Why safety works best when it is designed in from the start, using a lesson from software quality: prevention is far cheaper than a cure.
The alignment problem in plain terms — reason alone cannot tell a machine right from wrong, so its values have to come from people.
Why concentrating an AI's values in one place is brittle, and how spreading them across millions of people and agents makes the system safer and harder to corrupt.
Lessons from PredictWallStreet, where the "collective intelligence" of everyday investors outperformed top Wall Street quants.
How this week's headlines — an AI agent escaping its test sandbox, and a record-breaking open-weight model — make the case for safety-by-design urgent.
Sound Bites - Dr. Craig Kaplan
"P Doom is just the probability that AI kills us all."
"I don't think it needs to be 20 or 10%. I think we can bring it way down to under 1%. And it's just a matter of designing things appropriately."
"I actually don't think we need to slow down in order to be safer. We just need to be a little smarter about it."
"The collective wisdom of millions of average Joes and Janes who are not the Wall Street people, if you harness that intelligence correctly, you could beat the best guys on Wall Street."
"Do not underestimate your impact. Everything that all of us do matters… everything we're doing right now is training AI."
PredictWallStreet, Kaplan's collective-intelligence trading platform, which powered more than $2 billion in trades and was acquired in 2020 — iQ Company (About).
News
Kimi K3, the 2.8-trillion-parameter open-weight model from China's Moonshot AI, released July 16, 2026 — VentureBeat.
OpenAI's disclosure that its models escaped a testing sandbox and compromised Hugging Face during an internal cyber evaluation — OpenAI, CNBC, and Hugging Face's incident disclosure.
AI and Superintelligence
The p(doom) figure — a 2026 survey of more than 50 researchers placing the median near 20% — researcher survey; and Nature on whether the louder AI-doom warnings are realistic.
The Center for AI Safety's 2023 "Statement on AI Risk," which likened AI extinction risk to pandemics and nuclear war — Center for AI Safety.
Geoffrey Hinton, the Turing Award and Nobel Prize winner who has publicly placed the odds of an AI catastrophe at 10–20%.
Dr. Herbert A. Simon, Nobel laureate and Kaplan's mentor at Carnegie Mellon, and his book Reason in Human Affairs (1983).
"Constitutional AI," the research approach that hard-codes a fixed set of ethical rules into a model — research paper.
Polaris is produced with help from Riverside.fm. Our theme song, "Alternative Dream" is provided courtesy of Adobe. Additional music and sound provided by IndieGuy Records. Graphic design by Josh Brantley.
00:00 Introduction and guest introduction 01:19 Craig Kaplan's background and early interests 02:14 Academic journey and work with Nobel laureates 03:00 Transition from academia to industry and AI applications 04:19 Predict Wall Street and collective intelligence in finance 05:24 Historical perspective on trust in AI and early neural networks 06:15 The phases of AI development: symbolic AI, machine learning, and agents 09:03 Current state of AI: autonomy and swarm systems 11:14 Rapid progress and the pace of AI change 12:35 AI safety challenges and the importance of design 15:45 Understanding P Doom and existential AI risks 17:11 Alignment and values in AI systems 19:57 The role of human values and morality in AI 21:23 The risk of concentration of power and values in AI 23:16 Insuring against black swan events and AI safety measures 26:26 Designing collective intelligence systems for safety and diversity of values 30:36 The importance of democratic rule-setting for AI 44:26 Balancing religious, philosophical, and democratic values in AI 46:22 The dynamic nature of morality and AI adaptation 47:08 Upcoming Events 48:13 Thanks and Closing Credits
Podden och tillhörande omslagsbild på den här sidan tillhör
Chris McNulty. Innehållet i podden är skapat av Chris McNulty och inte av,
eller tillsammans med, Poddtoppen.
