RPO: Retrieval Preference Optimization for Robust Retrieval-Augmented Generation

TL;DR

This paper introduces Retrieval Preference Optimization (RPO), a novel alignment method for retrieval-augmented generation that improves response accuracy by adaptively leveraging multi-source knowledge based on retrieval relevance.

Contribution

RPO is the first RAG-specific alignment approach that explicitly quantifies retrieval relevance during training, enhancing response accuracy without additional components.

Findings

RPO outperforms existing RAG methods by 4-10% in accuracy.

RPO demonstrates robust generalization across four datasets.

It effectively integrates retrieval evaluation into response generation.

Abstract

While Retrieval-Augmented Generation (RAG) has exhibited promise in utilizing external knowledge, its generation process heavily depends on the quality and accuracy of the retrieved context. Large language models (LLMs) struggle to evaluate the correctness of non-parametric knowledge retrieved externally when it differs from internal memorization, leading to knowledge conflicts during response generation. To this end, we introduce the Retrieval Preference Optimization (RPO), a lightweight and effective alignment method to adaptively leverage multi-source knowledge based on retrieval relevance. An implicit representation of retrieval relevance is derived and incorporated into the reward model to integrate retrieval evaluation and response generation into a single model, solving the problem that previous methods necessitate the additional procedure to assess the retrieval quality. Notably, RPO is the only RAG-dedicated alignment approach that quantifies the awareness of retrieval relevance in training, overcoming mathematical obstacles. Experiments on four datasets demonstrate that RPO outperforms RAG by 4-10% in accuracy without any extra component, exhibiting its robust generalization.