CoRect: Context-Aware Logit Contrast for Hidden State Rectification to Resolve Knowledge Conflicts

TL;DR

CoRect introduces a novel method to rectify hidden states in retrieval-augmented generation models, effectively reducing hallucinations and improving faithfulness by contrasting contextualized and non-contextualized logits.

Contribution

It proposes a layer-wise logit contrast method that identifies and rectifies parametric biases without needing ground-truth labels, enhancing model faithfulness.

Findings

Improves faithfulness in QA and summarization tasks

Reduces hallucinations compared to baseline models

Effective across multiple benchmarks

Abstract

Retrieval-Augmented Generation (RAG) often struggles with knowledge conflicts, where model-internal parametric knowledge overrides retrieved evidence, leading to unfaithful outputs. Existing approaches are often limited, relying either on superficial decoding adjustments or weight editing that necessitates ground-truth targets. Through layer-wise analysis, we attribute this failure to a parametric suppression phenomenon: specifically, in deep layers, certain FFN layers overwrite context-sensitive representations with memorized priors. To address this, we propose CoRect (Context-Aware Logit Contrast for Hidden State Rectification). By contrasting logits from contextualized and non-contextualized forward passes, CoRect identifies layers that exhibit high parametric bias without requiring ground-truth labels. It then rectifies the hidden states to preserve evidence-grounded information. Across question answering (QA) and summarization benchmarks, CoRect consistently improves faithfulness and reduces hallucinations compared to strong baselines.