Understanding New-Knowledge-Induced Factual Hallucinations in LLMs: Analysis and Interpretation

TL;DR

This paper investigates how fine-tuning on new knowledge causes factual hallucinations in large language models, analyzing underlying mechanisms and proposing methods to mitigate such errors.

Contribution

It introduces a controlled dataset and provides a detailed analysis of hallucination mechanisms, highlighting the role of attention shifts and knowledge familiarity.

Findings

Hallucinations affect both new knowledge tasks and other evaluation tasks.

Familiarity of knowledge type influences hallucination severity more than the proportion of new knowledge.

Reintroducing known knowledge during training reduces hallucinations by restoring attention to key entities.

Abstract

Prior works have shown that fine-tuning on new knowledge can induce factual hallucinations in large language models (LLMs), leading to incorrect outputs when evaluated on previously known information. However, the specific manifestations of such hallucination and its underlying mechanisms remain insufficiently understood. Our work addresses this gap by designing a controlled dataset \textit{Biography-Reasoning}, and conducting a fine-grained analysis across multiple knowledge types and two task types, including knowledge question answering (QA) and knowledge reasoning tasks. We find that hallucinations not only severely affect tasks involving newly introduced knowledge, but also propagate to other evaluation tasks. Moreover, when fine-tuning on a dataset in which a specific knowledge type consists entirely of new knowledge, LLMs exhibit elevated hallucination tendencies. This suggests that the degree of unfamiliarity within a particular knowledge type, rather than the overall proportion of new knowledge, is a stronger driver of hallucinations. Through interpretability analysis, we show that learning new knowledge weakens the model's attention to key entities in the input question, leading to an over-reliance on surrounding context and a higher risk of hallucination. Conversely, reintroducing a small amount of known knowledge during the later stages of training restores attention to key entities and substantially mitigates hallucination behavior. Finally, we demonstrate that disrupted attention patterns can propagate across lexically similar contexts, facilitating the spread of hallucinations beyond the original task.