Evaluating the Retrieval Component in LLM-Based Question Answering Systems

TL;DR

This paper introduces a new evaluation framework for retrievers in LLM-based question answering systems, emphasizing alignment with overall system performance and accounting for LLMs' ability to ignore irrelevant information.

Contribution

It proposes a simple, effective baseline for assessing retrievers in RAG-based chatbots that better reflects their impact on QA system accuracy.

Findings

The evaluation method aligns well with overall QA performance.

Traditional metrics may not fully capture retriever effectiveness.

The framework considers LLMs' ability to ignore irrelevant contexts.

Abstract

Question answering systems (QA) utilizing Large Language Models (LLMs) heavily depend on the retrieval component to provide them with domain-specific information and reduce the risk of generating inaccurate responses or hallucinations. Although the evaluation of retrievers dates back to the early research in Information Retrieval, assessing their performance within LLM-based chatbots remains a challenge. This study proposes a straightforward baseline for evaluating retrievers in Retrieval-Augmented Generation (RAG)-based chatbots. Our findings demonstrate that this evaluation framework provides a better image of how the retriever performs and is more aligned with the overall performance of the QA system. Although conventional metrics such as precision, recall, and F1 score may not fully capture LLMs' capabilities - as they can yield accurate responses despite imperfect retrievers - our method considers LLMs' strengths to ignore irrelevant contexts, as well as potential errors and hallucinations in their responses.