Governing What You Cannot Observe: Adaptive Runtime Governance for Autonomous AI Agents

TL;DR

This paper introduces a theoretical framework and a practical system for adaptive runtime governance of autonomous AI agents, focusing on estimating and managing unobserved risks to ensure safety.

Contribution

It presents the Informational Viability Principle and the Agent Viability Framework, integrating Aubin's viability theory into a risk management system for autonomous agents.

Findings

Framework proven necessary and sufficient for failure mode coverage

RiskGate system instantiated with statistical estimators and safety pipelines

Transforming governance from reactive to predictive with Viability Index

Abstract

Autonomous AI agents can remain fully authorized and still become unsafe as behavior drifts, adversaries adapt, and decision patterns shift without any code change. We propose the \textbf{Informational Viability Principle}: governing an agent reduces to estimating a bound on unobserved risk $\hat{B}(x) = U(x) + SB(x) + RG(x)$ and allowing an action only when its capacity $S(x)$ exceeds $\hat{B}(x)$ by a safety margin. The \textbf{Agent Viability Framework}, grounded in Aubin's viability theory, establishes three properties -- monitoring (P1), anticipation (P2), and monotonic restriction (P3) -- as individually necessary and collectively sufficient for documented failure modes. \textbf{RiskGate} instantiates the framework with dedicated statistical estimators (KL divergence, segment-vs-rest $z$-tests, sequential pattern matching), a fail-secure monotonic pipeline, and a closed-loop Autopilot formalised as an instance of Aubin's regulation map with kill-switch-as-last-resort; a scalar Viability Index $VI(t) \in [-1,+1]$ with first-order $t^*$ prediction transforms governance from reactive to predictive. Contributions are the theoretical framework, the reference implementation, and analytical coverage against published agent-failure taxonomies; quantitative empirical evaluation is scoped as follow-up work.