KCR: Resolving Long-Context Knowledge Conflicts via Reasoning in LLMs

TL;DR

This paper introduces the KCR framework, which trains LLMs to resolve conflicting knowledge in long contexts by reinforcing correct reasoning paths through reinforcement learning, significantly improving their conflict resolution capabilities.

Contribution

The paper presents a novel reinforcement learning-based method to enhance LLMs' ability to resolve inter-context knowledge conflicts by guiding reasoning processes.

Findings

Significant improvement in conflict resolution accuracy

Enhanced reasoning consistency in long contexts

Effective training paradigm for LLMs to handle conflicting information

Abstract

Knowledge conflicts commonly arise across diverse sources, and their prevalence has increased with the advent of LLMs. When dealing with conflicts between multiple contexts, also known as \emph{inter-context knowledge conflicts}, LLMs are often confused by lengthy and conflicting contexts. To address this challenge, we propose the Knowledge Conflict Reasoning (KCR) framework, which enhances the ability of LLMs to resolve conflicting knowledge. The key idea of KCR is to train backbone LLMs to establish a correct reasoning process by rewarding them for selecting and adhering to the context with stronger logical consistency when presented with conflicting contexts. Specifically, we first extract reasoning paths, represented by either text or local knowledge graphs, from the conflicting long contexts. Subsequently, we employ Reinforcement Learning to encourage the model to learn the paradigm of reasoning process that follows correct reasoning paths rather than the incorrect counterparts. This enables the backbone models to genuinely acquire the capability to resolve inter-context knowledge conflicts within long contexts. Experimental results demonstrate that our framework significantly improves the ability of various backbone models to resolve knowledge conflicts in long-context scenarios, yielding substantial performance gains.