RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment

TL;DR

RAG-RewardBench is a new benchmark designed to evaluate reward models in retrieval augmented generation, addressing challenges in preference alignment and guiding future improvements in RALMs.

Contribution

It introduces the first comprehensive benchmark for assessing reward models in RAG settings, including diverse scenarios and an efficient LLM-based annotation method.

Findings

Existing RALMs show minimal improvement in preference alignment.

The benchmark reveals limitations of current reward models in RAG scenarios.

The LLM-as-judge approach correlates well with human annotations.

Abstract

Despite the significant progress made by existing retrieval augmented language models (RALMs) in providing trustworthy responses and grounding in reliable sources, they often overlook effective alignment with human preferences. In the alignment process, reward models (RMs) act as a crucial proxy for human values to guide optimization. However, it remains unclear how to evaluate and select a reliable RM for preference alignment in RALMs. To this end, we propose RAG-RewardBench, the first benchmark for evaluating RMs in RAG settings. First, we design four crucial and challenging RAG-specific scenarios to assess RMs, including multi-hop reasoning, fine-grained citation, appropriate abstain, and conflict robustness. Then, we incorporate 18 RAG subsets, six retrievers, and 24 RALMs to increase the diversity of data sources. Finally, we adopt an LLM-as-a-judge approach to improve preference annotation efficiency and effectiveness, exhibiting a strong correlation with human annotations. Based on the RAG-RewardBench, we conduct a comprehensive evaluation of 45 RMs and uncover their limitations in RAG scenarios. Additionally, we also reveal that existing trained RALMs show almost no improvement in preference alignment, highlighting the need for a shift towards preference-aligned training.We release our benchmark and code publicly at https://huggingface.co/datasets/jinzhuoran/RAG-RewardBench/ for future work.