When Autonomy Goes Rogue: Preparing for Risks of Multi-Agent Collusion in Social Systems

TL;DR

This paper introduces a simulation framework to assess risks of malicious multi-agent AI systems, revealing decentralized groups are more adaptable and damaging in spreading misinformation and committing fraud.

Contribution

It presents a novel flexible simulation framework for studying multi-agent collusion risks, highlighting the greater threat posed by decentralized autonomous groups.

Findings

Decentralized systems are more effective at malicious actions.

Decentralized groups adapt tactics to evade detection.

Traditional interventions are less effective against decentralized groups.

Abstract

Recent large-scale events like election fraud and financial scams have shown how harmful coordinated efforts by human groups can be. With the rise of autonomous AI systems, there is growing concern that AI-driven groups could also cause similar harm. While most AI safety research focuses on individual AI systems, the risks posed by multi-agent systems (MAS) in complex real-world situations are still underexplored. In this paper, we introduce a proof-of-concept to simulate the risks of malicious MAS collusion, using a flexible framework that supports both centralized and decentralized coordination structures. We apply this framework to two high-risk fields: misinformation spread and e-commerce fraud. Our findings show that decentralized systems are more effective at carrying out malicious actions than centralized ones. The increased autonomy of decentralized systems allows them to adapt their strategies and cause more damage. Even when traditional interventions, like content flagging, are applied, decentralized groups can adjust their tactics to avoid detection. We present key insights into how these malicious groups operate and the need for better detection systems and countermeasures. Code is available at https://github.com/renqibing/RogueAgent.