VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems

TL;DR

VerifyMAS introduces a hypothesis verification framework for failure attribution in LLM multi-agent systems, improving global failure detection and reducing search complexity through trajectory-level error validation.

Contribution

It proposes a novel verification-based approach that formulates failure hypotheses, enabling more accurate and efficient global failure attribution in multi-agent systems.

Findings

VerifyMAS outperforms prior methods on Aegis-Bench and Who&When datasets.

It improves failure attribution accuracy across various backbone models.

The framework reduces search space and maintains inference efficiency.

Abstract

Large language model-driven multi-agent systems (LLM-MAS) excel at complex tasks, yet unreliable agents remain a key bottleneck to system-level reliability. Automatic failure attribution is therefore critical, but existing approaches, such as direct prediction of agent-error pairs and agent-first failure attribution, rely on local logs of agents and miss global failures that only manifest over full interaction trajectories, such as cross-step inconsistencies and inter-agent coordination errors. Moreover, directly predicting failures induces a large combinatorial search space, hindering fine-grained attribution. To address these challenges, we propose VerifyMAS, a hypothesis verification framework for agent failure attribution. Instead of directly predicting faulty agents and error types, VerifyMAS formulates and verifies failure hypotheses against full trajectories. This verification-based approach decomposes attribution into trajectory-level error validation and fine-grained agent localization, providing an error-first attribution approach that captures global failure patterns while substantially reducing the search space. We further introduce a hypothesis-based data construction strategy grounded in a structured error taxonomy and fine-tune a specialized LLM verifier model for trajectory-level failure verification and agent attribution. Experiments on Aegis-Bench and Who&When show that VerifyMAS consistently improves diverse backbone models, including open-source Qwen and API-based GPT models, outperforming prior methods without sacrificing inference efficiency for long multi-agent trajectories.