Preventing Rogue Agents Improves Multi-Agent Collaboration

TL;DR

This paper introduces a monitoring approach to detect and prevent rogue agents in multi-agent systems, significantly improving collaboration success across various environments.

Contribution

It proposes a novel method for monitoring agents during action prediction to prevent failures caused by rogue agents in multi-agent systems.

Findings

Performance gains up to 17.4%, 2.5%, and 20% in different environments.

Effective identification of critical confusion points by monitors.

Interventions successfully prevent error propagation.

Abstract

Multi-agent systems, where specialized agents collaborate to solve a shared task hold great potential, from increased modularity to simulating complex environments. However, they also have a major caveat -- a single agent can cause the entire system to fail. Consider a simple game where the knowledge to solve the task is distributed between agents, which share information in a communication channel. At each round, any of the agents can terminate the game and make the final prediction, even if they are uncertain about the outcome of their action. Detection of such rogue agents before they act may prevent the system's failure. In this work, we propose to monitor agents during action prediction and intervene when a future error is likely to occur. To test our approach, we introduce WhoDunitEnv, a multi-agent collaboration environment that allows modular control over task complexity and communication structure. Experiments on WhoDunitEnv, code generation tasks and the GovSim environment for resource sustainability show that our approach leads to substantial performance gains up to 17.4%, 2.5% and 20%, respectively. Thorough analysis shows that our monitors successfully identify critical points of agent confusion and our interventions effectively stop agent errors from propagating.