PA-RAG: RAG Alignment via Multi-Perspective Preference Optimization

TL;DR

PA-RAG introduces a multi-perspective preference optimization method to improve retrieval-augmented generation, enhancing informativeness, robustness, and citation quality of large language models through fine-tuning and preference alignment.

Contribution

It proposes a novel multi-perspective preference alignment approach for RAG, combining SFT and DPO to better align generators with RAG requirements.

Findings

Significant performance improvements on four QA datasets.

Effective alignment with RAG requirements across multiple LLMs.

Enhanced response quality in informativeness and robustness.

Abstract

The emergence of Retrieval-augmented generation (RAG) has alleviated the issues of outdated and hallucinatory content in the generation of large language models (LLMs), yet it still reveals numerous limitations. When a general-purpose LLM serves as the RAG generator, it often suffers from inadequate response informativeness, response robustness, and citation quality. Past approaches to tackle these limitations, either by incorporating additional steps beyond generating responses or optimizing the generator through supervised fine-tuning (SFT), still failed to align with the RAG requirement thoroughly. Consequently, optimizing the RAG generator from multiple preference perspectives while maintaining its end-to-end LLM form remains a challenge. To bridge this gap, we propose Multiple Perspective Preference Alignment for Retrieval-Augmented Generation (PA-RAG), a method for optimizing the generator of RAG systems to align with RAG requirements comprehensively. Specifically, we construct high-quality instruction fine-tuning data and multi-perspective preference data by sampling varied quality responses from the generator across different prompt documents quality scenarios. Subsequently, we optimize the generator using SFT and Direct Preference Optimization (DPO). Extensive experiments conducted on four question-answer datasets across three LLMs demonstrate that PA-RAG can significantly enhance the performance of RAG generators. Our code and datasets are available at https://github.com/wujwyi/PA-RAG.