Affirmative safety: An approach to risk management for high-risk AI

TL;DR

This paper advocates for a proactive safety assurance approach for high-risk AI, requiring developers to demonstrate that their systems maintain risks below acceptable levels through technical and organizational evidence.

Contribution

It introduces the concept of affirmative safety cases for AI, outlining technical and operational evidence needed to support risk thresholds and proposing a framework aligned with existing risk management principles.

Findings

Proposes a new safety assurance framework for high-risk AI.

Identifies technical and organizational evidence types for safety cases.

Aligns AI safety practices with established high-risk industry standards.

Abstract

Prominent AI experts have suggested that companies developing high-risk AI systems should be required to show that such systems are safe before they can be developed or deployed. The goal of this paper is to expand on this idea and explore its implications for risk management. We argue that entities developing or deploying high-risk AI systems should be required to present evidence of affirmative safety: a proactive case that their activities keep risks below acceptable thresholds. We begin the paper by highlighting global security risks from AI that have been acknowledged by AI experts and world governments. Next, we briefly describe principles of risk management from other high-risk fields (e.g., nuclear safety). Then, we propose a risk management approach for advanced AI in which model developers must provide evidence that their activities keep certain risks below regulator-set thresholds. As a first step toward understanding what affirmative safety cases should include, we illustrate how certain kinds of technical evidence and operational evidence can support an affirmative safety case. In the technical section, we discuss behavioral evidence (evidence about model outputs), cognitive evidence (evidence about model internals), and developmental evidence (evidence about the training process). In the operational section, we offer examples of organizational practices that could contribute to affirmative safety cases: information security practices, safety culture, and emergency response capacity. Finally, we briefly compare our approach to the NIST AI Risk Management Framework. Overall, we hope our work contributes to ongoing discussions about national and global security risks posed by AI and regulatory approaches to address these risks.