MASPrism: Lightweight Failure Attribution for Multi-Agent Systems Using Prefill-Stage Signals

TL;DR

MASPrism is a lightweight failure attribution framework for multi-agent systems that uses prefill-stage signals from small language models to identify failure sources efficiently.

Contribution

It introduces a novel two-pass prefill-stage signal analysis method that improves failure source identification without costly workflows or training.

Findings

Achieves up to 33.41% improvement in Top-1 accuracy on Who&When-HC dataset.

Outperforms proprietary LLMs like Gemini-2.5-Pro on TRAIL dataset.

Processes traces in 2.66 seconds, 6.69 times faster than baseline.

Abstract

Failure attribution in LLM-based multi-agent systems aims to identify the steps that contribute to a failed execution. This task remains difficult because a single execution can contain many agent actions and tool calls, failure evidence can appear many steps after the original mistake, and existing methods often rely on costly agent workflows, replay, or training on synthetic failure logs. To address these challenges, we propose MASPrism, a lightweight framework that performs failure attribution using prefill-stage signals from a small language model (SLM). MASPrism first extracts token-level negative log-likelihood and attention weights during a prefill pass to identify symptom-like steps and earlier candidate sources, without decoding. It then reconstructs a focused diagnostic prompt and performs a second prefill pass to rank failure-source candidates. Using Qwen3-0.6B as the SLM, MASPrism achieves the best performance on three of the four evaluated subsets across Who&When and TRAIL, improving Top-1 accuracy on Who&When-HC by 33.41% over the best baseline. On TRAIL, MASPrism outperforms strong proprietary LLMs, including Gemini-2.5-Pro, with up to 89.50% relative improvement. MASPrism processes each trace in 2.66 seconds on average, achieving a 6.69$\times$ speedup over the single-pass prompting baseline, with zero output tokens. These results show that MASPrism provides an effective and practical framework for failure attribution in long multi-agent execution logs.