CAR: Query-Guided Confidence-Aware Reranking for Retrieval-Augmented Generation

TL;DR

CAR is a novel reranking framework that uses generator confidence changes to improve document selection for retrieval-augmented generation, enhancing answer quality across various datasets and models.

Contribution

It introduces a training-free, query-guided reranking method that leverages generator confidence to better select useful documents for RAG tasks.

Findings

CAR improves NDCG@5 across multiple datasets and retrievers.

It enhances the YesNo reranker by 25.4% on average with Contriever.

Ranking gains from CAR strongly correlate with downstream F1 improvements.

Abstract

Retrieval-Augmented Generation (RAG) depends on document ranking to provide useful evidence for generation, but conventional reranking methods mainly optimize query-document relevance rather than generation usefulness. A relevant document may still introduce noise, while a lower-ranked document may better reduce the generator's uncertainty. We propose CAR (Confidence-Aware Reranking), a query-guided, training-free, and plug-and-play reranking framework that uses generator confidence change as a document usefulness signal. CAR estimates confidence through the semantic consistency of multiple sampled answers under query-only and query-document conditions. Documents that significantly increase confidence are promoted, those that decrease confidence are demoted, and uncertain cases preserve the baseline order, while a query-level gate avoids unnecessary intervention on already confident queries. Experiments on four BEIR datasets show that CAR consistently improves NDCG@5 across sparse and dense retrievers, LLM-based and supervised rerankers, and four LLM backbones. Notably, CAR improves the YesNo reranker by 25.4 percent on average under Contriever retrieval, and its ranking gains strongly correlate with downstream generation F1 improvements, achieving Spearman rho = 0.964.