LURE-RAG: Lightweight Utility-driven Reranking for Efficient RAG

TL;DR

LURE-RAG introduces a lightweight, utility-driven reranking method for RAG that optimizes document order using listwise ranking loss, improving efficiency and effectiveness in downstream tasks.

Contribution

The paper presents LURE-RAG, a novel reranking framework that enhances retrieval utility in RAG through efficient training with listwise loss guided by LLM utility, requiring less resources.

Findings

Achieves 97-98% of state-of-the-art performance with greater efficiency.

Dense variant UR-RAG outperforms existing baselines by up to 3%.

Efficient training and inference in RAG with utility-driven reranking.

Abstract

Most conventional Retrieval-Augmented Generation (RAG) pipelines rely on relevance-based retrieval, which often misaligns with utility -- that is, whether the retrieved passages actually improve the quality of the generated text specific to a downstream task such as question answering or query-based summarization. The limitations of existing utility-driven retrieval approaches for RAG are that, firstly, they are resource-intensive typically requiring query encoding, and that secondly, they do not involve listwise ranking loss during training. The latter limitation is particularly critical, as the relative order between documents directly affects generation in RAG. To address this gap, we propose Lightweight Utility-driven Reranking for Efficient RAG (LURE-RAG), a framework that augments any black-box retriever with an efficient LambdaMART-based reranker. Unlike prior methods, LURE-RAG trains the reranker with a listwise ranking loss guided by LLM utility, thereby directly optimizing the ordering of retrieved documents. Experiments on two standard datasets demonstrate that LURE-RAG achieves competitive performance, reaching 97-98% of the state-of-the-art dense neural baseline, while remaining efficient in both training and inference. Moreover, its dense variant, UR-RAG, significantly outperforms the best existing baseline by up to 3%.