Ragas: Automated Evaluation of Retrieval Augmented Generation

TL;DR

Ragas is a reference-free evaluation framework for RAG systems that assesses retrieval quality, faithful passage utilization, and generation quality, facilitating faster development of retrieval-augmented language models.

Contribution

Introduces Ragas, a suite of metrics for evaluating RAG systems without ground truth annotations, addressing multiple evaluation dimensions.

Findings

Provides metrics for retrieval relevance and focus

Assesses faithfulness in passage utilization

Evaluates generation quality without human annotations

Abstract

We introduce Ragas (Retrieval Augmented Generation Assessment), a framework for reference-free evaluation of Retrieval Augmented Generation (RAG) pipelines. RAG systems are composed of a retrieval and an LLM based generation module, and provide LLMs with knowledge from a reference textual database, which enables them to act as a natural language layer between a user and textual databases, reducing the risk of hallucinations. Evaluating RAG architectures is, however, challenging because there are several dimensions to consider: the ability of the retrieval system to identify relevant and focused context passages, the ability of the LLM to exploit such passages in a faithful way, or the quality of the generation itself. With Ragas, we put forward a suite of metrics which can be used to evaluate these different dimensions \textit{without having to rely on ground truth human annotations}. We posit that such a framework can crucially contribute to faster evaluation cycles of RAG architectures, which is especially important given the fast adoption of LLMs.