RADIANT: Retrieval AugmenteD entIty-context AligNmenT -- Introducing RAG-ability and Entity-Context Divergence

TL;DR

This paper introduces Radiant, a framework that improves the factual accuracy of retrieval-augmented language models by aligning retrieved evidence with generated responses, addressing challenges in entity retention and context fidelity.

Contribution

It proposes Radiant, a novel alignment framework that enhances RAG performance by integrating retrieval with generation more effectively, extending DPO for better factual consistency.

Findings

RAG-ability is generally low across LLMs, indicating significant challenges in factual integration.

Radiant significantly improves entity retention and factual accuracy in retrieval-augmented generation.

The proposed method reduces hallucinations and enhances context fidelity in generated content.

Abstract

As Large Language Models (LLMs) continue to advance, Retrieval-Augmented Generation (RAG) has emerged as a vital technique to enhance factual accuracy by integrating external knowledge into the generation process. However, LLMs often fail to faithfully integrate retrieved evidence into their generated responses, leading to factual inconsistencies. To quantify this gap, we introduce Entity-Context Divergence (ECD), a metric that measures the extent to which retrieved information is accurately reflected in model outputs. We systematically evaluate contemporary LLMs on their ability to preserve factual consistency in retrieval-augmented settings, a capability we define as RAG-ability. Our empirical analysis reveals that RAG-ability remains low across most LLMs, highlighting significant challenges in entity retention and context fidelity. This paper introduces Radiant (Retrieval AugmenteD entIty-context AligNmenT), a novel framework that merges RAG with alignment designed to optimize the interplay between retrieved evidence and generated content. Radiant extends Direct Preference Optimization (DPO) to teach LLMs how to integrate provided additional information into subsequent generations. As a behavior correction mechanism, Radiant boosts RAG performance across varied retrieval scenarios, such as noisy web contexts, knowledge conflicts, and hallucination reduction. This enables more reliable, contextually grounded, and factually coherent content generation.