MaMa: A Game-Theoretic Approach for Designing Safe Agentic Systems

TL;DR

This paper introduces MaMa, a game-theoretic framework using LLMs to automatically design multi-agent systems that remain safe against adversarial compromises, ensuring robustness and safety in diverse environments.

Contribution

The paper presents MaMa, a novel algorithm that formulates safe system design as a Stackelberg game and employs LLM-based adversarial search to enhance safety against worst-case attacks.

Findings

MaMa systems effectively defend against worst-case adversarial attacks.

Designed systems maintain high task performance while ensuring safety.

Robustness extends to different adversaries and attack objectives.

Abstract

LLM-based multi-agent systems have demonstrated impressive capabilities, but they also introduce significant safety risks when individual agents fail or behave adversarially. In this work, we study the automated design of agentic systems that remain safe even when a subset of agents is compromised. We formalize this challenge as a Stackelberg security game between a system designer (the Meta-Agent) and a best-responding Meta-Adversary that selects and compromises a subset of agents to minimize safety. We propose Meta-Adversary-Meta-Agent (MaMa), a novel algorithm for approximately solving this game and automatically designing safe agentic systems. Our approach uses LLM-based adversarial search, where the Meta-Agent iteratively proposes system designs and receives feedback based on the strongest attacks discovered by the Meta-Adversary. Empirical evaluations across diverse environments show that systems designed with MaMa consistently defend against worst-case attacks while maintaining performance comparable to systems optimized solely for task success. Moreover, the resulting systems generalize to stronger adversaries, as well as ones with different attack objectives or underlying LLMs, demonstrating robust safety beyond the training setting.