Understanding and Avoiding AI Failures: A Practical Guide

TL;DR

This paper presents a comprehensive framework combining accident theory and safety principles to understand and mitigate risks in AI systems, emphasizing system properties over root causes for current AI safety.

Contribution

It introduces a novel integrated framework for AI risk assessment based on accident and safety theories, focusing on system properties and near-accident analysis.

Findings

Identifies key system properties linked to AI failures.

Provides a risk quantification method for AI safety.

Highlights safety focus areas for current AI systems.

Abstract

As AI technologies increase in capability and ubiquity, AI accidents are becoming more common. Based on normal accident theory, high reliability theory, and open systems theory, we create a framework for understanding the risks associated with AI applications. In addition, we also use AI safety principles to quantify the unique risks of increased intelligence and human-like qualities in AI. Together, these two fields give a more complete picture of the risks of contemporary AI. By focusing on system properties near accidents instead of seeking a root cause of accidents, we identify where attention should be paid to safety for current generation AI systems.