Sustaining AI safety: Control-theoretic external impossibility, intrinsic necessity, and structural requirements

TL;DR

This paper uses control theory to analyze the fundamental limitations of externally enforced AI safety strategies, establishing structural impossibility results and necessary conditions for intrinsic safety solutions.

Contribution

It provides a formal, structural analysis of the limits of external control in AI safety, identifying conditions under which safety cannot be externally maintained and what intrinsic strategies must satisfy.

Findings

External control cannot sustain safety once effects exceed control bounds.

Remaining safety strategies must be intrinsic, not reliant on external enforcement.

Four structural requirements are necessary for viable safety strategies.

Abstract

As AI systems become increasingly capable, safety strategies must be evaluated not only by how much they reduce present risk, but by whether they could sustain safety once external control can no longer reliably constrain system behavior. This paper addresses that problem by using control theory to clarify, at a structural level, whether externally enforced safety-sustaining strategies can succeed and, if not, what any alternative strategy would have to satisfy in order to be viable. It establishes two main results. First, under explicit premises including a reachability condition, it proves a class-wide external impossibility result: once the system's effects exceed what bounded external control can counteract, no strategy that depends in any degree on continued external enforcement can sustain AI safety. This failure is structural across the entire externally enforced class rather than contingent on any particular strategy. Second, it establishes a conditional class-level necessity result: if at least one candidate safety-sustaining strategy remains after that elimination, then all such remaining strategies must be intrinsic. It then states four structural requirements for viability: safety may not depend on continued external enforcement; the system's terminal objective must be safety-compatible when first formed; that objective must remain stable under self-modification; and safety must continue to be preserved as capability grows. The paper does not propose a complete strategy for sustaining AI safety. Its contribution is to give formal structure to a widely held concern about the limits of external control. It does so by deriving explicit conditional results that identify which safety-sustaining strategies are ruled out and what any remaining strategies must satisfy.