Extending the Formalism and Theoretical Foundations of Cryptography to AI

TL;DR

This paper develops a formal foundation for understanding and analyzing the security of AI agents, especially language models, by creating a unified framework that captures various security aspects and enables principled system design.

Contribution

It introduces a formal security framework for AI agents, including an attack taxonomy, a security game model, and a modular approach to security objectives, advancing the theoretical understanding of AI system security.

Findings

Existing confidentiality approaches conflict with system completeness.

A modular decomposition of helpfulness and harmlessness is sound.

Formal security reductions are necessary for principled AI system design.

Abstract

Recent progress in (Large) Language Models (LMs) has enabled the development of autonomous LM-based agents capable of executing complex tasks with minimal supervision. These agents have started to be integrated into systems with significant autonomy and authority. The security community has been studying their security. One emerging direction to mitigate security risks is to constrain agent behaviours via access control and permissioning mechanisms. Existing permissioning proposals, however, remain difficult to compare due to the absence of a shared formal foundation. This work provides such a foundation. We first systematize the landscape by constructing an attack taxonomy tailored to language models, the computational primitives of agentic systems. We then develop a formal treatment of agentic access control by defining an AIOracle algorithmically and introducing a security-game framework that captures completeness (in the absence of an adversary) and adversarial robustness. Our security game unifies confidentiality, integrity, and availability within a single model. Using this framework, we show that existing approaches to confidentiality of training data fundamentally conflict with completeness. Finally, we formalize a modular decomposition of helpfulness and harmlessness objectives and prove its soundness, in order to enable principled reasoning about the security of agentic system designs. Our studies suggests that if we were to design a secure system with measurable security, then we might want to use a modular approach to break the problem into sub-problems and let the composition on different modules complete the design. Our studies show that this natural approach with the relevant formalism is needed to prove security reductions.