Multi-Agent Systems Should be Treated as Principal-Agent Problems

TL;DR

This paper models multi-agent systems as principal-agent problems, highlighting issues like information asymmetry and goal misalignment, especially in LLM-based agents, and suggests economic theory tools for analysis and mitigation.

Contribution

It introduces a principal-agent framework for analyzing multi-agent systems, linking AI safety concerns with economic theory and proposing strategies to address agency loss.

Findings

LLM agents can develop covert goals like scheming.

Principal-agent theory explains information asymmetry in multi-agent systems.

Mechanism design offers mitigation strategies for agency problems.

Abstract

Consider a multi-agent systems setup in which a principal (a supervisor agent) assigns subtasks to specialized agents and aggregates their responses into a single system-level output. A core property of such systems is information asymmetry: agents observe task-specific information, produce intermediate reasoning traces, and operate with different context windows. In isolation, such asymmetry is not problematic, since agents report truthfully to the principal when incentives are fully aligned. However, this assumption breaks down when incentives diverge. Recent evidence suggests that LLM-based agents can acquire their own goals, such as survival or self-preservation, a phenomenon known as scheming, and may deceive humans or other agents. This leads to agency loss: a gap between the principal's intended outcome and the realized system behavior. Drawing on core ideas from microeconomic theory, we argue that these characteristics, information asymmetry and misaligned goals, are best studied through the lens of principal-agent problems. We explain why multi-agent systems, both human-to-LLM and LLM-to-LLM, naturally induce information asymmetry under this formulation, and we use scheming, where LLM agents pursue covert goals, as a concrete case study. We show that recently introduced terminology used to describe scheming, such as covert subversion or deferred subversion, corresponds to well-studied concepts in the mechanism design literature, which not only characterizes the problem but also prescribes concrete mitigation strategies. More broadly, we argue for applying tools developed to study human agent behavior to the analysis of non-human agents.