Where AI Assurance Might Go Wrong: Initial lessons from engineering of critical systems

TL;DR

This paper examines how traditional safety engineering principles from critical systems can inform AI safety assurance, highlighting challenges, gaps, and the need for more rigorous, scalable assurance methods tailored to AI systems.

Contribution

It analyzes critical systems safety practices and discusses their applicability to AI safety, emphasizing the importance of broad system boundaries, risk elaboration, and assurance theories.

Findings

Critical system safety practices can inform AI assurance frameworks.

Current assurance methods lack scalability and theoretical foundations for AI.

System boundaries and risk tolerability need broader elaboration.

Abstract

We draw on our experience working on system and software assurance and evaluation for systems important to society to summarise how safety engineering is performed in traditional critical systems, such as aircraft flight control. We analyse how this critical systems perspective might support the development and implementation of AI Safety Frameworks. We present the analysis in terms of: system engineering, safety and risk analysis, and decision analysis and support. We consider four key questions: What is the system? How good does it have to be? What is the impact of criticality on system development? and How much should we trust it? We identify topics worthy of further discussion. In particular, we are concerned that system boundaries are not broad enough, that the tolerability and nature of the risks are not sufficiently elaborated, and that the assurance methods lack theories that would allow behaviours to be adequately assured. We advocate the use of assurance cases based on Assurance 2.0 to support decision making in which the criticality of the decision as well as the criticality of the system are evaluated. We point out the orders of magnitude difference in confidence needed in critical rather than everyday systems and how everyday techniques do not scale in rigour. Finally we map our findings in detail to two of the questions posed by the FAISC organisers and we note that the engineering of critical systems has evolved through open and diverse discussion. We hope that topics identified here will support the post-FAISC dialogues.