CORD: Balancing COnsistency and Rank Distillation for Robust Retrieval-Augmented Generation

TL;DR

CORD introduces a novel training method for retrieval-augmented generation that balances consistency regularization with rank-aware distillation, improving robustness across various benchmarks.

Contribution

It proposes an adaptive sampling approach to balance consistency and rank distillation, enhancing RAG model robustness.

Findings

Outperforms existing methods on diverse RAG benchmarks

Balances consistency and rank preservation effectively

Improves generation robustness and relevance

Abstract

With the adoption of retrieval-augmented generation (RAG), large language models (LLMs) are expected to ground their generation to the retrieved contexts. Yet, this is hindered by position bias of LLMs, failing to evenly attend to all contexts. Previous work has addressed this by synthesizing contexts with perturbed positions of gold segment, creating a position-diversified train set. We extend this intuition to propose consistency regularization with augmentation and distillation. First, we augment each training instance with its position perturbation to encourage consistent predictions, regardless of ordering. We also distill behaviors of this pair, although it can be counterproductive in certain RAG scenarios where the given order from the retriever is crucial for generation quality. We thus propose CORD, balancing COnsistency and Rank Distillation. CORD adaptively samples noise-controlled perturbations from an interpolation space, ensuring both consistency and respect for the rank prior. Empirical results show this balance enables CORD to outperform consistently in diverse RAG benchmarks.