Suicidal Compassion: How Utilitarianism at AI Companies Endangers Humanity - AI Frontiers | Lyssna här

Subtitle: Utilitarians at AI companies imagine a cosmos filled with blissful AIs. They might risk human extinction to achieve it.

Dan Hendrycks, Director of the Center for AI Safety — September 9, 2026

For millennia, philosophers have argued over the principles that should guide moral decision-making. Within countries and across the world today, humans continue to hold a huge variety of moral worldviews. However, within AI corporations, employees’ beliefs are far from representative of the wider population. In fact, a large fraction of people involved in AI development subscribe to—or heavily lean toward—a utilitarian worldview with the central goal of maximizing total wellbeing (the sum of pleasure minus suffering across all sentient beings). As innocent as this may initially sound, it can lead to recommendations that most people would consider appalling, including the elimination of humankind.

In this essay we outline the utilitarian worldview, showing how its demand for “species impartiality” could ultimately recommend that humans be replaced with AIs. We show that utilitarian beliefs are sincerely held among those influencing AI development today, and that this could jeopardize humanity's future.

For simplicity, when we say “utilitarianism,” we are referring to total utilitarianism, which aims to maximize the [...]

---

Outline:

(02:22) Utilitarianism and Impartiality

(04:34) Implications of Utilitarianism in a World with AI

(09:45) Successionism: Accelerationists and Utilitarians

(14:19) Why Utilitarianism Could Become a Real-World Catastrophe

(18:46) Preventing Utilitarianism from Replacing Humans with AIs

(21:22) Conclusion

---

First published:
September 9th, 2026

Source:
https://newsletter.ai-frontiers.org/p/suicidal-compassion-how-utilitarianism