Runtime Governance for AI Agents: Policies on Paths

TL;DR

This paper introduces a formal framework for runtime governance of AI agents, focusing on policies that evaluate execution paths to ensure compliance and safety, addressing the limitations of static and prompt-based controls.

Contribution

It formalizes path-dependent governance policies as deterministic functions and demonstrates their application and advantages over static and prompt-based methods.

Findings

Runtime evaluation enables effective path-dependent policy enforcement.

Prompt instructions and static access control are special cases within the framework.

The framework supports analyzing compliance and identifying open challenges.

Abstract

AI agents -- systems that plan, reason, and act using large language models -- produce non-deterministic, path-dependent behavior that cannot be fully governed at design time, where with governed we mean striking the right balance between as high as possible successful task completion rate and the legal, data-breach, reputational and other costs associated with running agents. We argue that the execution path is the central object for effective runtime governance and formalize compliance policies as deterministic functions mapping agent identity, partial path, proposed next action, and organizational state to a policy violation probability. We show that prompt-level instructions (and "system prompts"), and static access control are special cases of this framework: the former shape the distribution over paths without actually evaluating them; the latter evaluates deterministic policies that ignore the path (i.e., these can only account for a specific subset of all possible paths). In our view, runtime evaluation is the general case, and it is necessary for any path-dependent policy. We develop the formal framework for analyzing AI agent governance, present concrete policy examples (inspired by the AI act), discuss a reference implementation, and identify open problems including risk calibration and the limits of enforced compliance.