THELMA: Task Based Holistic Evaluation of Large Language Model Applications-RAG Question Answering

TL;DR

THELMA is a comprehensive, reference-free evaluation framework for RAG-based question answering systems, enabling detailed assessment and targeted improvements without reliance on labeled data.

Contribution

It introduces six novel metrics for holistic evaluation of RAG QA applications, facilitating fine-grained analysis and improvement guidance.

Findings

Metrics help identify specific RAG components needing enhancement

Framework supports monitoring and improving end-to-end RAG pipelines

Evaluation is possible without labeled sources or reference responses

Abstract

We propose THELMA (Task Based Holistic Evaluation of Large Language Model Applications), a reference free framework for RAG (Retrieval Augmented generation) based question answering (QA) applications. THELMA consist of six interdependent metrics specifically designed for holistic, fine grained evaluation of RAG QA applications. THELMA framework helps developers and application owners evaluate, monitor and improve end to end RAG QA pipelines without requiring labelled sources or reference responses.We also present our findings on the interplay of the proposed THELMA metrics, which can be interpreted to identify the specific RAG component needing improvement in QA applications.