Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarks

TL;DR

DREAM is a multi-agent debate framework using LLMs for more accurate IR benchmark annotation, significantly reducing human effort and uncovering missing relevant data to improve IR system evaluation.

Contribution

The paper introduces DREAM, a novel multi-round debate-based relevance assessment method that enhances labeling accuracy and uncovers missing data in IR benchmarks.

Findings

Achieved 95.2% labeling accuracy with only 3.5% human involvement.

Uncovered 29,824 missing relevant chunks in IR benchmarks.

Re-benchmarking shows unaddressed data holes distort IR system rankings.

Abstract

Information retrieval (IR) evaluation remains challenging due to incomplete IR benchmark datasets that contain unlabeled relevant chunks. While LLMs and LLM-human hybrid strategies reduce costly human effort, they remain prone to LLM overconfidence and ineffective AI-to-human escalation. To address this, we propose DREAM, a multi-round debate-based relevance assessment framework with LLM agents, built on opposing initial stances and iterative reciprocal critique. Through our agreement-based debate, it yields more accurate labeling for certain cases and more reliable AI-to-human escalation for uncertain ones, achieving 95.2% labeling accuracy with only 3.5% human involvement. Using DREAM, we build BRIDGE, a refined benchmark that mitigates evaluation bias and enables fairer retriever comparison by uncovering 29,824 missing relevant chunks. We then re-benchmark IR systems and extend evaluation to RAG, showing that unaddressed holes not only distort retriever rankings but also drive retrieval-generation misalignment. The relevance assessment framework is available at https: //github.com/DISL-Lab/DREAM-ICLR-26; and the BRIDGE dataset is available at https://github.com/DISL-Lab/BRIDGE-Benchmark.