SciRerankBench: Benchmarking Rerankers Towards Scientific Retrieval-Augmented Generated LLMs

TL;DR

This paper introduces SciRerankBench, a benchmark for evaluating rerankers in scientific retrieval-augmented LLMs, focusing on noise resilience, relevance, and factual accuracy across multiple scientific domains.

Contribution

It presents the first dedicated benchmark for reranker evaluation in scientific RAG-LLMs, including diverse question-context-answer pairs and systematic analysis of 13 rerankers across five scientific subjects.

Findings

Rerankers show varied strengths in noise resilience and relevance disambiguation.

Certain rerankers excel in factual consistency but struggle with noisy contexts.

Insights guide future development of more robust rerankers for scientific LLM applications.

Abstract

Scientific literature question answering is a pivotal step towards new scientific discoveries. Recently, \textit{two-stage} retrieval-augmented generated large language models (RAG-LLMs) have shown impressive advancements in this domain. Such a two-stage framework, especially the second stage (reranker), is particularly essential in the scientific domain, where subtle differences in terminology may have a greatly negative impact on the final factual-oriented or knowledge-intensive answers. Despite this significant progress, the potential and limitations of these works remain unexplored. In this work, we present a Scientific Rerank-oriented RAG Benchmark (SciRerankBench), for evaluating rerankers within RAG-LLMs systems, spanning five scientific subjects. To rigorously assess the reranker performance in terms of noise resilience, relevance disambiguation, and factual consistency, we develop three types of question-context-answer (Q-C-A) pairs, i.e., Noisy Contexts (NC), Semantically Similar but Logically Irrelevant Contexts (SSLI), and Counterfactual Contexts (CC). Through systematic evaluation of 13 widely used rerankers on five families of LLMs, we provide detailed insights into their relative strengths and limitations. To the best of our knowledge, SciRerankBench is the first benchmark specifically developed to evaluate rerankers within RAG-LLMs, which provides valuable observations and guidance for their future development.