Big Bang, Low Bar -- Risk Assessment in the Public Arena

TL;DR

This paper emphasizes the importance of rigorous risk assessment in public discourse, highlighting how neglecting the principle that unlikely but catastrophic risks should not be ignored can lead to dangerous oversight, especially in AI safety debates.

Contribution

It critically examines the neglect of fundamental risk management principles in public discussions, with a focus on existential AI risks, and advocates for more careful risk evaluation.

Findings

Public discourse often overlooks low-probability, high-impact risks.

Recent AI risk debates exemplify neglect of risk management principles.

Better risk assessment can prevent catastrophic oversights.

Abstract

One of the basic principles of risk management is that we should always keep an eye on ways that things could go badly wrong, even if they seem unlikely. The more disastrous a potential failure, the more improbable it needs to be, before we can safely ignore it. This principle may seem obvious, but it is easily overlooked in public discourse about risk, even by well-qualified commentators who should certainly know better. The present piece is prompted by neglect of the principle in recent discussions about the potential existential risks of artificial intelligence. The failing is not peculiar to this case, but recent debates in this area provide some particularly stark examples of how easily the principle can be overlooked.