Efficient Leave-one-out Approximation in LLM Multi-agent Debate Based on Introspection

TL;DR

This paper introduces IntrospecLOO, a novel, efficient method for approximating leave-one-out evaluations in LLM multi-agent debates, reducing computational costs while accurately assessing individual agent contributions.

Contribution

It proposes a simple prompting strategy that approximates leave-one-out analysis with fewer queries, enhancing evaluation efficiency in LLM multi-agent systems.

Findings

IntrospecLOO achieves comparable accuracy to traditional LOO methods.

The method significantly reduces query complexity and computational costs.

Experimental results on benchmark datasets validate its effectiveness.

Abstract

Multi-agent systems based on large language models (LLMs) advance automatic task completion in various fields, where debate is a common cooperation form for agents to solve complicated problems with reasoning and cross-review to solidify answers. Assessing the individual contributions of agents within these debates is crucial for system refinement and outcome reliability. Traditional leave-one-out (LOO) method offers a clear framework for evaluating each agent's role but face challenges in LLM-based systems due to high computational costs and associated financial implications. This paper presents introspective-leave-one-out (IntrospecLOO), a simple yet effective prompting for approximation of LOO in LLM-powered multi-agent debates. IntrospecLOO introduces an additional querying round after standard debates, prompting agents to update their answers while ignoring responses from a designated agent. This strategy effectively isolates and gauges each participant's influence at a reduced query complexity compared to the original LOO approaches. Validation through experiments on three benchmark datasets confirms the effectiveness of IntrospecLOO.