CTRL-RAG: Contrastive Likelihood Reward Based Reinforcement Learning for Context-Faithful RAG Models

TL;DR

This paper introduces CTRL-RAG, a reinforcement learning framework that enhances context-faithful retrieval-augmented generation models by using a contrastive likelihood reward to improve evidence extraction and answer confidence.

Contribution

It proposes a novel contrastive likelihood reward mechanism for RAG models, addressing faithfulness and hallucination issues without relying solely on external rewards.

Findings

Improves performance on faithfulness benchmarks

Reduces hallucinations in generated responses

Enhances evidence grounding in RAG models

Abstract

With the growing use of Retrieval-Augmented Generation (RAG), training large language models (LLMs) for context-sensitive reasoning and faithfulness is increasingly important. Existing RAG-oriented reinforcement learning (RL) methods rely on external rewards that often fail to evaluate document faithfulness, and may misjudge similar answers in open-domain settings. In addition, there is no RAG-based selfreward mechanism. Moreover, although such a mechanism could in principle estimate answer confidence given documents, the absence of objective feedback in a self-judgment can cause hallucination accumulation and eventual model collapse. To tackle these issues, we propose a novel "internal-external" hybrid reward framework centered on a Contrastive Likelihood Reward (CLR). CLR directly optimizes the log-likelihood gap between responses conditioned on prompts with and without supporting evidence. This encourages the model to extract relevant evidence and increases its confidence when grounded in a specific context. Experiments show that our method (used alone or combined with external correctness rewards) achieves strong performance on singlehop, multi-hop, vertical-domain, and faithfulness benchmarks. Our training code and models are coming soon.