Problem-Solving in Language Model Networks

TL;DR

This paper explores multi-agent debate on various network topologies to improve LLM reasoning, showing that network structure, consensus, and bias significantly influence question-answering accuracy.

Contribution

It extends multi-agent debate to general networks, analyzing how topology, influence, and bias affect collective reasoning in LLM systems.

Findings

Random networks perform similarly to fully connected networks with fewer tokens.

Strong consensus correlates with correct answers.

Biased hub nodes can enhance system performance.

Abstract

To improve the reasoning and question-answering capabilities of Large Language Models (LLMs), several multi-agent approaches have been introduced. While these methods enhance performance, the application of collective intelligence-based approaches to complex network structures and the dynamics of agent interactions remain underexplored. This work extends the concept of multi-agent debate to more general network topologies, measuring the question-answering accuracy, influence, consensus, and the effects of bias on the collective. The results show that random networks perform similarly to fully connected networks despite using significantly fewer tokens. Furthermore, a strong consensus among agents correlates with correct answers, whereas divided responses typically indicate incorrect answers. Analysing the influence of the agents reveals a balance between self-reflection and interconnectedness; self-reflection aids when local interactions are incorrect, and local interactions aid when the agent itself is incorrect. Additionally, bias plays a strong role in system performance with correctly biased hub nodes boosting performance. These insights suggest that using random networks or scale-free networks with knowledgeable agents placed in central positions can enhance the overall question-answering performance of multi-agent systems.