DeepEra: A Deep Evidence Reranking Agent for Scientific Retrieval-Augmented Generated Question Answering

TL;DR

DeepEra introduces a step-by-step reasoning based reranking method for scientific retrieval, significantly improving the logical relevance and factual grounding of retrieved passages in scientific QA tasks.

Contribution

It proposes DeepEra, a novel evidence reranking agent that enhances retrieval precision by evaluating logical relevance, and introduces SciRAG-SSLI, a large dataset for systematic evaluation of semantic similarity versus logical irrelevance.

Findings

DeepEra outperforms existing rerankers in retrieval tasks.

The SciRAG-SSLI dataset effectively tests logical robustness.

Empirical validation shows improved factual grounding in scientific QA.

Abstract

With the rapid growth of scientific literature, scientific question answering (SciQA) has become increasingly critical for exploring and utilizing scientific knowledge. Retrieval-Augmented Generation (RAG) enhances LLMs by incorporating knowledge from external sources, thereby providing credible evidence for scientific question answering. But existing retrieval and reranking methods remain vulnerable to passages that are semantically similar but logically irrelevant, often reducing factual reliability and amplifying hallucinations.To address this challenge, we propose a Deep Evidence Reranking Agent (DeepEra) that integrates step-by-step reasoning, enabling more precise evaluation of candidate passages beyond surface-level semantics. To support systematic evaluation, we construct SciRAG-SSLI (Scientific RAG - Semantically Similar but Logically Irrelevant), a large-scale dataset comprising about 300K SciQA instances across 10 subjects, constructed from 10M scientific corpus. The dataset combines naturally retrieved contexts with systematically generated distractors to test logical robustness and factual grounding. Comprehensive evaluations confirm that our approach achieves superior retrieval performance compared to leading rerankers. To our knowledge, this work is the first to comprehensively study and empirically validate innegligible SSLI issues in two-stage RAG frameworks.