Evaluation of RAG Metrics for Question Answering in the Telecom Domain

TL;DR

This paper evaluates RAG metrics for question answering in the telecom domain, modifies an existing evaluation library, and analyzes the effectiveness and challenges of these metrics in specialized, real-world scenarios.

Contribution

It introduces a modified RAGAS evaluation package with additional metrics and provides an analysis of their performance and limitations in the telecom domain.

Findings

Some metrics correlate with correct retrievals.

Domain adaptation affects metric scores.

Challenges exist in applying these metrics in real-world telecom QA.

Abstract

Retrieval Augmented Generation (RAG) is widely used to enable Large Language Models (LLMs) perform Question Answering (QA) tasks in various domains. However, RAG based on open-source LLM for specialized domains has challenges of evaluating generated responses. A popular framework in the literature is the RAG Assessment (RAGAS), a publicly available library which uses LLMs for evaluation. One disadvantage of RAGAS is the lack of details of derivation of numerical value of the evaluation metrics. One of the outcomes of this work is a modified version of this package for few metrics (faithfulness, context relevance, answer relevance, answer correctness, answer similarity and factual correctness) through which we provide the intermediate outputs of the prompts by using any LLMs. Next, we analyse the expert evaluations of the output of the modified RAGAS package and observe the challenges of using it in the telecom domain. We also study the effect of the metrics under correct vs. wrong retrieval and observe that few of the metrics have higher values for correct retrieval. We also study for differences in metrics between base embeddings and those domain adapted via pre-training and fine-tuning. Finally, we comment on the suitability and challenges of using these metrics for in-the-wild telecom QA task.