BracketRank: Large Language Model Document Ranking via Reasoning-based Competitive Elimination

TL;DR

BracketRank introduces a reasoning-based tournament framework for document ranking with large language models, significantly improving retrieval performance on complex reasoning benchmarks.

Contribution

It presents a novel bracket-style elimination method with reasoning prompts, enhancing LLM-based reranking beyond surface-level matching.

Findings

Achieves 26.56 nDCG@10 on BRIGHT benchmark, outperforming RankGPT-4 and Rank-R1-14B.

Surpasses baselines on TREC datasets with 77.90 and 75.85 nDCG@5 scores.

Demonstrates the effectiveness of explicit reasoning in competitive document reranking.

Abstract

Reasoning-intensive retrieval requires deep semantic inference beyond surface-level keyword matching, posing a challenge for current LLM-based rerankers limited by context constraints and order sensitivity. We propose \textbf{\BracketRank}, a framework that treats document reranking as a reasoning-driven competitive tournament. Our approach introduces three key innovations: (1) adaptive grouping based on model context limits, (2) reasoning-enhanced prompts that mandate step-by-step relevance explanations, and (3) a bracket-style elimination structure with winner and loser tracks. This design ensures robust document advancement while enabling parallel processing across competition stages. Evaluation on the BRIGHT reasoning benchmark shows that \BracketRank achieves \textbf{26.56 nDCG@10}, significantly outperforming state-of-the-art baselines including RankGPT-4 (17.0) and Rank-R1-14B (20.5). On TREC datasets, BracketRank achieves 77.90 nDCG@5 on DL 19 and 75.85 nDCG@5 on DL 20, exceeding all baselines, establishing that explicit reasoning within competitive elimination is a powerful paradigm for complex, multi-step retrieval tasks. https://github.com/DataScienceUIBK/BracketRank