Hit-RAG: Learning to Reason with Long Contexts via Preference Alignment

TL;DR

Hit-RAG introduces a multi-stage preference alignment framework that improves reasoning with long contexts in retrieval-augmented models by systematically refining evidence utilization, leading to significant performance gains across benchmarks.

Contribution

The paper presents a novel multi-stage preference alignment method that enhances long-context reasoning in retrieval-augmented models, addressing attention dilution and hallucination issues.

Findings

Consistently improves performance on eight benchmarks.

Outperforms larger models in long-context reasoning.

Effectively mitigates attention dilution and hallucinations.

Abstract

Despite the promise of Retrieval-Augmented Generation in grounding Multimodal Large Language Models with external knowledge, the transition to extensive contexts often leads to significant attention dilution and reasoning hallucinations. The surge in information density causes critical evidence to be submerged by voluminous noise, which complicates the discernment of relevant fragments within a dense input. In this paper, we propose \textbf{Hit-RAG}, a multi-stage preference alignment framework designed to resolve these cognitive bottlenecks through a progressive optimization pipeline. Our approach systematically refines the utilization of external evidence via three distinct stages. First, Supervised Fine-tuning establishes baseline context awareness to minimize information neglect. Next, Discriminative Preference Alignment enhances robustness against misleading distractors. Finally, Group-Relative Policy Optimization stabilizes logical synthesis to prevent reasoning collapse. Extensive evaluations on eight benchmarks demonstrate that Hit-RAG consistently yields substantial performance gains, enabling models to bridge the gap between context acquisition and accurate reasoning while surpassing much larger counterparts in long-context scenarios.