RAGVUE: A Diagnostic View for Explainable and Automated Evaluation of Retrieval-Augmented Generation

TL;DR

RAGVUE is a comprehensive, explainable framework for automated evaluation of RAG systems, providing detailed insights into retrieval, reasoning, and grounding errors to improve understanding and development.

Contribution

It introduces RAGVUE, a novel diagnostic tool that decomposes RAG performance into multiple explainable metrics with structured explanations, supporting manual and automated evaluation.

Findings

RAGVUE uncovers fine-grained failures overlooked by existing metrics.

It offers a transparent, explainable evaluation process for RAG systems.

The framework is publicly available with APIs and interfaces for easy integration.

Abstract

Evaluating Retrieval-Augmented Generation (RAG) systems remains a challenging task: existing metrics often collapse heterogeneous behaviors into single scores and provide little insight into whether errors arise from retrieval,reasoning, or grounding. In this paper, we introduce RAGVUE, a diagnostic and explainable framework for automated, reference-free evaluation of RAG pipelines. RAGVUE decomposes RAG behavior into retrieval quality, answer relevance and completeness, strict claim-level faithfulness, and judge calibration. Each metric includes a structured explanation, making the evaluation process transparent. Our framework supports both manual metric selection and fully automated agentic evaluation. It also provides a Python API, CLI, and a local Streamlit interface for interactive usage. In comparative experiments, RAGVUE surfaces fine-grained failures that existing tools such as RAGAS often overlook. We showcase the full RAGVUE workflow and illustrate how it can be integrated into research pipelines and practical RAG development. The source code and detailed instructions on usage are publicly available on GitHub