"Merge Conflicts!" Exploring the Impacts of External Distractors to Parametric Knowledge Graphs

TL;DR

This paper investigates how large language models' responses are affected by external distractors, revealing risks of hallucination and knowledge interference during interactions, through a systematic framework and knowledge graph analysis.

Contribution

It introduces a framework to systematically study the impact of external knowledge on LLMs and constructs a parametric knowledge graph to analyze knowledge structure disruptions.

Findings

LLMs' responses deviate from parametric knowledge when faced with conflicts.

External distractors can cause LLMs to hallucinate or produce incorrect responses.

LLMs are sensitive to the truthfulness of external knowledge but can still be misled by unrelated information.

Abstract

Large language models (LLMs) acquire extensive knowledge during pre-training, known as their parametric knowledge. However, in order to remain up-to-date and align with human instructions, LLMs inevitably require external knowledge during their interactions with users. This raises a crucial question: How will LLMs respond when external knowledge interferes with their parametric knowledge? To investigate this question, we propose a framework that systematically elicits LLM parametric knowledge and introduces external knowledge. Specifically, we uncover the impacts by constructing a parametric knowledge graph to reveal the different knowledge structures of LLMs, and introduce external knowledge through distractors of varying degrees, methods, positions, and formats. Our experiments on both black-box and open-source models demonstrate that LLMs tend to produce responses that deviate from their parametric knowledge, particularly when they encounter direct conflicts or confounding changes of information within detailed contexts. We also find that while LLMs are sensitive to the veracity of external knowledge, they can still be distracted by unrelated information. These findings highlight the risk of hallucination when integrating external knowledge, even indirectly, during interactions with current LLMs. All the data and results are publicly available.