Multi-agent decision making: A Blackwell's informativeness approach

TL;DR

This paper analyzes multi-agent decision-making with large language models using Blackwell's informativeness framework, establishing theoretical bounds and proposing a practical posterior pooling method that improves QA performance.

Contribution

It introduces a formal analysis of multi-LLM decision structures and develops a novel posterior pooling method for improved question-answering accuracy.

Findings

Voting and debate are less informative than pooled private information.

Bayesian pooled posterior maximization serves as an optimal decision rule.

The proposed method outperforms existing multi-LLM debate and voting approaches on six benchmarks.

Abstract

The rapid development of large language models (LLMs) has motivated research on decision-making in multi-agent systems, where multiple agents collaborate to achieve shared objectives. Existing aggregation approaches, such as voting and debate, are largely ad-hoc and lack formal guarantees regarding the informativeness of the resulting decisions. In this paper, we provide a principled approach to analyse decisions made in the multi-LLM setting using Blackwell's informativeness framework. Within the Blackwell information-structure abstraction, we show that voting and debate induce information structures that are no more informative than the pooled private information of all agents. This result identifies Bayesian pooled posterior maximisation as an information-theoretic upper-bound decision rule under the Blackwell ordering. Motivated by this theoretical analysis, we introduce a practical method for LLM-based question-answering (QA) tasks that estimates each agent's posterior and approximates the pooled posterior using a product-of-posteriors estimator. Extensive experiments on six QA benchmarks demonstrate that our approach outperforms state-of-the-art multi-LLM debate and voting methods.