Safety is Non-Compositional: A Formal Framework for Capability-Based AI Systems

TL;DR

This paper proves formally that safety properties in capability-based AI systems are non-compositional, meaning combined agents can reach forbidden goals even if individual agents cannot, due to emergent conjunctive dependencies.

Contribution

It introduces the first formal proof demonstrating the non-compositional nature of safety in AI systems with conjunctive capability dependencies.

Findings

Safety is non-compositional in conjunctive capability scenarios

Two agents individually incapable of reaching forbidden goals can do so jointly

Emergent conjunctive dependencies enable forbidden capabilities

Abstract

This paper contains the first formal proof that safety is non-compositional in the presence of conjunctive capability dependencies: two agents each individually inca- pable of reaching any forbidden capability can, when combined, collectively reach a forbidden goal through an emergent conjunctive dependency.