InspectorRAGet: An Introspection Platform for RAG Evaluation

TL;DR

InspectorRAGet is a comprehensive, publicly available platform designed for detailed analysis and evaluation of RAG system outputs, integrating human and algorithmic metrics for improved assessment.

Contribution

It introduces a novel introspection platform that enables detailed, multi-level analysis of RAG systems, addressing the lack of comprehensive evaluation tools.

Findings

Supports aggregate and instance-level analysis

Incorporates human and algorithmic metrics

Available publicly for community use

Abstract

Large Language Models (LLM) have become a popular approach for implementing Retrieval Augmented Generation (RAG) systems, and a significant amount of effort has been spent on building good models and metrics. In spite of increased recognition of the need for rigorous evaluation of RAG systems, few tools exist that go beyond the creation of model output and automatic calculation. We present InspectorRAGet, an introspection platform for performing a comprehensive analysis of the quality of RAG system output. InspectorRAGet allows the user to analyze aggregate and instance-level performance of RAG systems, using both human and algorithmic metrics as well as annotator quality. InspectorRAGet is suitable for multiple use cases and is available publicly to the community. A live instance of the platform is available at https://ibm.biz/InspectorRAGet.