Every Response Counts: Quantifying Uncertainty of LLM-based Multi-Agent Systems through Tensor Decomposition

TL;DR

This paper introduces MATU, a tensor decomposition-based framework that quantifies uncertainty in multi-agent systems by analyzing entire reasoning trajectories, addressing challenges in communication variability and role dependencies.

Contribution

The paper presents a novel tensor decomposition approach for uncertainty quantification in multi-agent systems, capturing complex interactions beyond final outputs.

Findings

MATU effectively estimates uncertainty across diverse tasks.

It disentangles sources of uncertainty in multi-agent communication.

The framework generalizes across different agent structures.

Abstract

While Large Language Model-based Multi-Agent Systems (MAS) consistently outperform single-agent systems on complex tasks, their intricate interactions introduce critical reliability challenges arising from communication dynamics and role dependencies. Existing Uncertainty Quantification methods, typically designed for single-turn outputs, fail to address the unique complexities of the MAS. Specifically, these methods struggle with three distinct challenges: the cascading uncertainty in multi-step reasoning, the variability of inter-agent communication paths, and the diversity of communication topologies. To bridge this gap, we introduce MATU, a novel framework that quantifies uncertainty through tensor decomposition. MATU moves beyond analyzing final text outputs by representing entire reasoning trajectories as embedding matrices and organizing multiple execution runs into a higher-order tensor. By applying tensor decomposition, we disentangle and quantify distinct sources of uncertainty, offering a comprehensive reliability measure that is generalizable across different agent structures. We provide comprehensive experiments to show that MATU effectively estimates holistic and robust uncertainty across diverse tasks and communication topologies.