DiscoUQ: Structured Disagreement Analysis for Uncertainty Quantification in LLM Agent Ensembles

TL;DR

DiscoUQ introduces a novel framework that analyzes inter-agent disagreement in multi-agent LLM systems to produce better uncertainty estimates, leveraging linguistic and embedding structures for improved calibration and robustness.

Contribution

The paper presents DiscoUQ, a structured disagreement analysis framework that enhances uncertainty quantification in LLM ensembles by utilizing semantic and geometric disagreement features.

Findings

DiscoUQ-LLM achieves an average AUROC of 0.802, outperforming baselines.

DiscoUQ provides well-calibrated confidence estimates with lower ECE.

Features generalize across diverse benchmarks with minimal performance loss.

Abstract

Multi-agent LLM systems, where multiple prompted instances of a language model independently answer questions, are increasingly used for complex reasoning tasks. However, existing methods for quantifying the uncertainty of their collective outputs rely on shallow voting statistics that discard the rich semantic information in agents' reasoning. We introduce DiscoUQ, a framework that extracts and leverages the structure of inter-agent disagreement -- both linguistic properties (evidence overlap, argument strength, divergence depth) and embedding geometry (cluster distances, dispersion, cohesion) -- to produce well-calibrated confidence estimates. We propose three methods of increasing complexity: DiscoUQ-LLM (logistic regression on LLM-extracted structure features), DiscoUQ-Embed (logistic regression on embedding geometry), and DiscoUQ-Learn (a neural network combining all features). Evaluated on four diverse benchmarks (StrategyQA, MMLU, TruthfulQA, ARC-Challenge) with a 5-agent system using Qwen3.5-27B, DiscoUQ-LLM achieves an average AUROC of 0.802, outperforming the best baseline (LLM Aggregator, 0.791) while being substantially better calibrated (ECE 0.036 vs. 0.098). The learned features generalize across benchmarks with near-zero performance degradation and provide the largest improvements where they are most needed: in the ambiguous "weak disagreement" tier where simple vote counting fails.