Open-Source Reproduction and Explainability Analysis of Corrective Retrieval Augmented Generation

TL;DR

This paper presents an open-source reproduction of CRAG, a system that enhances retrieval-augmented generation robustness, and provides the first explainability analysis revealing its reliance on named entity alignment.

Contribution

We provide a fully open-source implementation of CRAG replacing proprietary components and analyze its retrieval evaluator using SHAP for explainability.

Findings

Open-source pipeline achieves comparable performance to original CRAG.

SHAP analysis shows the evaluator relies mainly on named entity alignment.

Identifies domain transfer limitations on science questions.

Abstract

Corrective Retrieval Augmented Generation (CRAG) improves the robustness of RAG systems by evaluating retrieved document quality and triggering corrective actions. However, the original implementation relies on proprietary components including the Google Search API and closed model weights, limiting reproducibility. In this work, we present a fully open-source reproduction of CRAG, replacing proprietary web search with the Wikipedia API and the original LLaMA-2 generator with Phi-3-mini-4k-instruct. We evaluate on PopQA and ARC-Challenge, demonstrating that our open-source pipeline achieves comparable performance to the original system. Furthermore, we contribute the first explainability analysis of CRAG's T5-based retrieval evaluator using SHAP, revealing that the evaluator primarily relies on named entity alignment rather than semantic similarity. Our analysis identifies key failure modes including domain transfer limitations on science questions. All code and results are available at https://github.com/suryayalavarthi/crag-reproduction.