LANCER: LLM Reranking for Nugget Coverage

TL;DR

LANCER is a novel LLM-based reranking method designed to improve document retrieval coverage for long-form report generation by predicting sub-questions and ranking documents accordingly.

Contribution

It introduces a new reranking approach that focuses on maximizing nugget coverage, addressing limitations of relevance-focused retrieval methods.

Findings

LANCER achieves higher nugget coverage metrics than existing reranking methods.

Sub-question generation significantly improves retrieval quality.

Empirical results demonstrate enhanced information coverage in retrieval tasks.

Abstract

Unlike short-form retrieval-augmented generation (RAG), such as factoid question answering, long-form RAG requires retrieval to provide documents covering a wide range of relevant information. Automated report generation exemplifies this setting: it requires not only relevant information but also a more elaborate response with comprehensive information. Yet, existing retrieval methods are primarily optimized for relevance ranking rather than information coverage. To address this limitation, we propose LANCER, an LLM-based reranking method for nugget coverage. LANCER predicts what sub-questions should be answered to satisfy an information need, predicts which documents answer these sub-questions, and reranks documents in order to provide a ranked list covering as many information nuggets as possible. Our empirical results show that LANCER enhances the quality of retrieval as measured by nugget coverage metrics, achieving higher $\alpha$-nDCG and information coverage than other LLM-based reranking methods. Our oracle analysis further reveals that sub-question generation plays an essential role.