ReasoningRank: Teaching Student Models to Rank through Reasoning-Based Knowledge Distillation

TL;DR

Reason-to-Rank (R2R) introduces an open-source, transparent reranking method that uses large language models to generate explanations, improving accuracy and interpretability in document relevance ranking.

Contribution

The paper presents R2R, a novel approach that distills reasoning-based explanations from large language models into smaller, interpretable reranking models for information retrieval.

Findings

R2R achieves competitive reranking performance on MSMARCO and BRIGHT datasets.

The approach enhances transparency by providing reasoning explanations.

R2R improves both accuracy and interpretability in document reranking.

Abstract

Reranking documents based on their relevance to a given query is a critical task in information retrieval. Traditional reranking methods often lack transparency and rely on proprietary models, hindering reproducibility and interpretability. We propose Reason-to-Rank (R2R), a novel open-source reranking approach that enhances transparency by generating two types of reasoning: direct relevance reasoning, which explains how a document addresses the query, and comparison reasoning, which justifies the relevance of one document over another. We leverage large language models (LLMs) as teacher models to generate these explanations and distill this knowledge into smaller, openly available student models. Our student models are trained to generate meaningful reasoning and rerank documents, achieving competitive performance across multiple datasets, including MSMARCO and BRIGHT. Experiments demonstrate that R2R not only improves reranking accuracy but also provides valuable insights into the decision-making process. By offering a structured and interpretable solution with openly accessible resources, R2R aims to bridge the gap between effectiveness and transparency in information retrieval, fostering reproducibility and further research in the field.