Collaborative QA using Interacting LLMs. Impact of Network Structure, Node Capability and Distributed Data

TL;DR

This paper models and analyzes how networks of interacting LLMs perform collaborative question-answering, focusing on hallucination spread and the impact of network structure, node capability, and data distribution.

Contribution

It introduces a novel generative model combining mean-field dynamics and a randomized utility model to analyze LLM interactions and hallucination diffusion.

Findings

Hallucination spread increases with network connectivity and data heterogeneity.

Node capability and incentive levels significantly influence truthfulness in LLM networks.

Network structure and framing affect the accuracy and reliability of collaborative QA.

Abstract

In this paper, we model and analyze how a network of interacting LLMs performs collaborative question-answering (CQA) in order to estimate a ground truth given a distributed set of documents. This problem is interesting because LLMs often hallucinate when direct evidence to answer a question is lacking, and these effects become more pronounced in a network of interacting LLMs. The hallucination spreads, causing previously accurate LLMs to hallucinate. We study interacting LLMs and their hallucination by combining novel ideas of mean-field dynamics (MFD) from network science and the randomized utility model from economics to construct a useful generative model. We model the LLM with a latent state that indicates if it is truthful or not with respect to the ground truth, and extend a tractable analytical model considering an MFD to model the diffusion of information in a directed network of LLMs. To specify the probabilities that govern the dynamics of the MFD, we propose a randomized utility model. For a network of LLMs, where each LLM has two possible latent states, we posit sufficient conditions for the existence and uniqueness of a fixed point and analyze the behavior of the fixed point in terms of the incentive (e.g., test-time compute) given to individual LLMs. We experimentally study and analyze the behavior of a network of $100$ open-source LLMs with respect to data heterogeneity, node capability, network structure, and sensitivity to framing on multiple semi-synthetic datasets.