What are the odds? Risk and uncertainty about AI existential risk

TL;DR

This paper critiques linear risk models for AI existential threats, emphasizing the importance of accounting for epistemic indifference and different types of uncertainty to better estimate the probability of catastrophic outcomes.

Contribution

It highlights the limitations of existing risk models and introduces a nuanced discussion of uncertainty types affecting AI existential risk assessments.

Findings

Linear models may underestimate risk due to structural assumptions.

Epistemic indifference can increase perceived probability of catastrophic outcomes.

Distinguishing between risk and uncertainty improves understanding of AI existential threats.

Abstract

This work is a commentary of the article \href{https://doi.org/10.18716/ojs/phai/2025.2801}{AI Survival Stories: a Taxonomic Analysis of AI Existential Risk} by Cappelen, Goldstein, and Hawthorne. It is not just a commentary though, but a useful reminder of the philosophical limitations of \say{linear} models of risk. The article will focus on the model employed by the authors: first, I discuss some differences between standard Swiss Cheese models and this one. I then argue that in a situation of epistemic indifference the probability of P(D) is higher than what one might first suggest, given the structural relationships between layers. I then distinguish between risk and uncertainty, and argue that any estimation of P(D) is structurally affected by two kinds of uncertainty: option uncertainty and state-space uncertainty. Incorporating these dimensions of uncertainty into our qualitative discussion on AI existential risk can provide a better understanding of the likeliness of P(D).