LimRank: Less is More for Reasoning-Intensive Information Reranking

TL;DR

This paper introduces LIMRANK, a reasoning-intensive reranker trained with minimal supervision and synthetic data, achieving competitive results with significantly less training data and computational resources.

Contribution

The authors propose LIMRANK and LIMRANK-SYNTHESIZER, enabling effective reranking with minimal supervision and open-source synthetic data generation, reducing reliance on large-scale fine-tuning.

Findings

LIMRANK performs competitively on reasoning-intensive benchmarks.

Training with less than 5% of typical data yields strong results.

LIMRANK generalizes well across various downstream tasks.

Abstract

Existing approaches typically rely on large-scale fine-tuning to adapt LLMs for information reranking tasks, which is computationally expensive. In this work, we demonstrate that modern LLMs can be effectively adapted using only minimal, high-quality supervision. To enable this, we design LIMRANK-SYNTHESIZER, a reusable and open-source pipeline for generating diverse, challenging, and realistic reranking examples. Using this synthetic data, we fine-tune our reranker model, LIMRANK. We evaluate LIMRANK on two challenging benchmarks, i.e., BRIGHT for reasoning-intensive retrieval and FollowIR for instruction-following retrieval. Our experiments demonstrate that LIMRANK achieves competitive performance, while being trained on less than 5% of the data typically used in prior work. Further ablation studies demonstrate the effectiveness of LIMRANK-SYNTHESIZER and the strong generalization capabilities of LIMRANK across downstream tasks, including scientific literature search and retrieval-augmented generation for knowledge-intensive problem solving.