ASRank: Zero-Shot Re-Ranking with Answer Scent for Document Retrieval

TL;DR

ASRank introduces a zero-shot re-ranking method using answer scent scoring with large language models, significantly improving document retrieval accuracy in open-domain question answering tasks.

Contribution

The paper presents a novel zero-shot re-ranking approach based on answer scent, enhancing retrieval quality without additional training.

Findings

ASRank improves Top-1 accuracy on NQ from 19.2% to 46.5%.

ASRank outperforms state-of-the-art methods like UPR on multiple datasets.

The approach is effective across diverse question answering datasets.

Abstract

Retrieval-Augmented Generation (RAG) models have drawn considerable attention in modern open-domain question answering. The effectiveness of RAG depends on the quality of the top retrieved documents. However, conventional retrieval methods sometimes fail to rank the most relevant documents at the top. In this paper, we introduce ASRank, a new re-ranking method based on scoring retrieved documents using zero-shot answer scent which relies on a pre-trained large language model to compute the likelihood of the document-derived answers aligning with the answer scent. Our approach demonstrates marked improvements across several datasets, including NQ, TriviaQA, WebQA, ArchivalQA, HotpotQA, and Entity Questions. Notably, ASRank increases Top-1 retrieval accuracy on NQ from $19.2\%$ to $46.5\%$ for MSS and $22.1\%$ to $47.3\%$ for BM25. It also shows strong retrieval performance on several datasets compared to state-of-the-art methods (47.3 Top-1 by ASRank vs 35.4 by UPR by BM25).