Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation Strategies

TL;DR

This paper compares evaluation strategies for multi-hop reasoning in retrieval-augmented generation systems, introducing CARE, a context-aware method that outperforms existing approaches especially in complex multi-hop scenarios.

Contribution

The paper proposes CARE, a novel context-aware evaluation strategy for RAG systems, demonstrating its effectiveness over existing methods in multi-hop reasoning tasks.

Findings

CARE outperforms existing evaluation methods for multi-hop reasoning.

Performance gains are larger in models with more parameters and longer contexts.

Single-hop query evaluation shows minimal sensitivity to context-aware methods.

Abstract

Retrieval-augmented generation (RAG) enhances large language models (LLMs) with external knowledge to answer questions more accurately. However, research on evaluating RAG systems-particularly the retriever component-remains limited, as most existing work focuses on single-context retrieval rather than multi-hop queries, where individual contexts may appear irrelevant in isolation but are essential when combined. In this research, we use the HotPotQA, MuSiQue, and SQuAD datasets to simulate a RAG system and compare three LLM-as-judge evaluation strategies, including our proposed Context-Aware Retriever Evaluation (CARE). Our goal is to better understand how multi-hop reasoning can be most effectively evaluated in RAG systems. Experiments with LLMs from OpenAI, Meta, and Google demonstrate that CARE consistently outperforms existing methods for evaluating multi-hop reasoning in RAG systems. The performance gains are most pronounced in models with larger parameter counts and longer context windows, while single-hop queries show minimal sensitivity to context-aware evaluation. Overall, the results highlight the critical role of context-aware evaluation in improving the reliability and accuracy of retrieval-augmented generation systems, particularly in complex query scenarios. To ensure reproducibility, we provide the complete data of our experiments at https://github.com/lorenzbrehme/CARE.