Free Energy Risk Metrics for Systemically Safe AI: Gatekeeping Multi-Agent Study

TL;DR

This paper introduces a novel risk metric based on the Free Energy Principle for multi-agent AI systems, enabling transparent risk governance and demonstrating safety improvements in autonomous vehicle simulations.

Contribution

It proposes a Cumulative Risk Exposure metric grounded in the Free Energy Principle, offering a flexible, transparent framework for risk assessment in multi-agent AI systems.

Findings

Gatekeepers improve collective safety even at low penetration.

The framework accounts for uncertainty in world and preference models.

Demonstrated in autonomous vehicle simulations with positive safety externalities.

Abstract

We investigate the Free Energy Principle as a foundation for measuring risk in agentic and multi-agent systems. From these principles we introduce a Cumulative Risk Exposure metric that is flexible to differing contexts and needs. We contrast this to other popular theories for safe AI that hinge on massive amounts of data or describing arbitrarily complex world models. In our framework, stakeholders need only specify their preferences over system outcomes, providing straightforward and transparent decision rules for risk governance and mitigation. This framework naturally accounts for uncertainty in both world model and preference model, allowing for decision-making that is epistemically and axiologically humble, parsimonious, and future-proof. We demonstrate this novel approach in a simplified autonomous vehicle environment with multi-agent vehicles whose driving policies are mediated by gatekeepers that evaluate, in an online fashion, the risk to the collective safety in their neighborhood, and intervene through each vehicle's policy when appropriate. We show that the introduction of gatekeepers in an AV fleet, even at low penetration, can generate significant positive externalities in terms of increased system safety.