STAMP/STPA Informed Characterization of Factors Leading to Loss of Control in AI Systems

TL;DR

This paper introduces a structured framework based on STAMP/STPA to analyze and identify factors leading to loss of control in AI systems, addressing safety concerns across current and future AI developments.

Contribution

It adapts the STAMP/STPA safety analysis methodology to the AI domain, providing a new approach for characterizing and mitigating loss of control risks in AI systems.

Findings

Framework helps identify causal factors of control loss in AI

Application of STAMP/STPA clarifies control structure failures

Supports safer design of socio-technical AI systems

Abstract

A major concern amongst AI safety practitioners is the possibility of loss of control, whereby humans lose the ability to exert control over increasingly advanced AI systems. The range of concerns is wide, spanning current day risks to future existential risks, and a range of loss of control pathways from rapid AI self-exfiltration scenarios to more gradual disempowerment scenarios. In this work we set out to firstly, provide a more structured framework for discussing and characterizing loss of control and secondly, to use this framework to assist those responsible for the safe operation of AI-containing socio-technical systems to identify causal factors leading to loss of control. We explore how these two needs can be better met by making use of a methodology developed within the safety-critical systems community known as STAMP and its associated hazard analysis technique of STPA. We select the STAMP methodology primarily because it is based around a world-view that socio-technical systems can be functionally modeled as control structures, and that safety issues arise when there is a loss of control in these structures.