The Subtle Art of Defection: Understanding Uncooperative Behaviors in LLM based Multi-Agent Systems

TL;DR

This paper presents a framework for simulating and analyzing uncooperative behaviors in LLM-based multi-agent systems, revealing their potential to cause rapid system collapse and evaluating detection methods.

Contribution

It introduces a game theory-based taxonomy and a multi-stage simulation pipeline for uncooperative behaviors in multi-agent systems, filling a key research gap.

Findings

Uncooperative behaviors can cause system collapse within 1-7 rounds.

LLM-based detection methods detect some behaviors but miss others.

Cooperative agents maintain perfect stability, uncooperative ones cause rapid failure.

Abstract

This paper introduces a novel framework for simulating and analyzing how uncooperative behaviors can destabilize or collapse LLM-based multi-agent systems. Our framework includes two key components: (1) a game theory-based taxonomy of uncooperative agent behaviors, addressing a notable gap in the existing literature; and (2) a structured, multi-stage simulation pipeline that dynamically generates and refines uncooperative behaviors as agents' states evolve. We evaluate the framework via a collaborative resource management setting, measuring system stability using metrics such as survival time and resource overuse rate. Empirically, our framework achieves 96.7% accuracy in generating realistic uncooperative behaviors, validated by human evaluations. Our results reveal a striking contrast: cooperative agents maintain perfect system stability (100% survival over 12 rounds with 0% resource overuse), while any uncooperative behavior can trigger rapid system collapse within 1 to 7 rounds. We also evaluate LLM-based defense methods, finding they detect some uncooperative behaviors, but some behaviors remain largely undetectable. These gaps highlight how uncooperative agents degrade collective outcomes and underscore the need for more resilient multi-agent systems.