Rank4Gen: RAG-Preference-Aligned Document Set Selection and Ranking

TL;DR

Rank4Gen is a generator-aware document ranking model for RAG that optimizes for response quality and accounts for generator-specific preferences, improving evidence selection and response accuracy.

Contribution

It introduces a generator-specific ranking approach and a new dataset, PRISM, to align document ranking with generator preferences and response quality in RAG systems.

Findings

Rank4Gen outperforms existing models on five RAG benchmarks.

It effectively models generator-specific preferences for evidence ranking.

The PRISM dataset enables training and evaluation of generator-aware rankers.

Abstract

In the RAG paradigm, the information retrieval module provides context for generators by retrieving and ranking multiple documents to support the aggregation of evidence. However, existing ranking models are primarily optimized for query--document relevance, which often misaligns with generators' preferences for evidence selection and citation, limiting their impact on response quality. Moreover, most approaches do not account for preference differences across generators, resulting in unstable cross-generator performance. We propose \textbf{Rank4Gen}, a generator-aware ranker for RAG that targets the goal of \emph{Ranking for Generators}. Rank4Gen introduces two key preference modeling strategies: (1) \textbf{From Ranking Relevance to Response Quality}, which optimizes ranking with respect to downstream response quality rather than query--document relevance; and (2) \textbf{Generator-Specific Preference Modeling}, which conditions a single ranker on different generators to capture their distinct ranking preferences. To enable such modeling, we construct \textbf{PRISM}, a dataset built from multiple open-source corpora and diverse downstream generators. Experiments on five challenging and recent RAG benchmarks demonstrate that Rank4Gen achieves strong and competitive performance for complex evidence composition in RAG.