Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluation

TL;DR

This paper introduces a new multi-perspective approach to failure attribution in multi-agent systems, along with a benchmark and evaluation protocol, revealing limitations of prior methods and emphasizing the importance of considering attribution ambiguity.

Contribution

It proposes multi-perspective failure attribution, introduces MP-Bench for evaluation, and demonstrates the need for multi-perspective benchmarks in MAS failure analysis.

Findings

Prior conclusions about LLMs struggling with failure attribution are influenced by benchmark limitations.

Multi-perspective benchmarks reveal more realistic failure attribution challenges.

The new benchmark and protocol improve the evaluation of MAS failure diagnosis methods.

Abstract

Failure attribution is essential for diagnosing and improving multi-agent systems (MAS), yet existing benchmarks and methods largely assume a single deterministic root cause for each failure. In practice, MAS failures often admit multiple plausible attributions due to complex inter-agent dependencies and ambiguous execution trajectories. We revisit MAS failure attribution from a multi-perspective standpoint and propose multi-perspective failure attribution, a practical paradigm that explicitly accounts for attribution ambiguity. To support this setting, we introduce MP-Bench, the first benchmark designed for multi-perspective failure attribution in MAS, along with a new evaluation protocol tailored to this paradigm. Through extensive experiments, we find that prior conclusions suggesting LLMs struggle with failure attribution are largely driven by limitations in existing benchmark designs. Our results highlight the necessity of multi-perspective benchmarks and evaluation protocols for realistic and reliable MAS debugging.