Unraveling Misinformation Propagation in LLM Reasoning

TL;DR

This paper investigates how misinformation affects reasoning in Large Language Models, revealing that misinformation significantly degrades accuracy and that early correction strategies and fine-tuning can mitigate this issue.

Contribution

It provides a detailed analysis of misinformation propagation in LLM reasoning and proposes effective mitigation strategies such as early correction and fine-tuning.

Findings

Misinformation causes accuracy drops of 10.02% to 72.20%.

LLMs succeed less than half the time in correcting misinformation.

Early factual corrections and fine-tuning improve reasoning accuracy.

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities in reasoning, positioning them as promising tools for supporting human problem-solving. However, what happens when their performance is affected by misinformation, i.e., incorrect inputs introduced by users due to oversights or gaps in knowledge? Such misinformation is prevalent in real-world interactions with LLMs, yet how it propagates within LLMs' reasoning process remains underexplored. Focusing on mathematical reasoning, we present a comprehensive analysis of how misinformation affects intermediate reasoning steps and final answers. We also examine how effectively LLMs can correct misinformation when explicitly instructed to do so. Even with explicit instructions, LLMs succeed less than half the time in rectifying misinformation, despite possessing correct internal knowledge, leading to significant accuracy drops (10.02% - 72.20%), and the degradation holds with thinking models (4.30% - 19.97%). Further analysis shows that applying factual corrections early in the reasoning process most effectively reduces misinformation propagation, and fine-tuning on synthesized data with early-stage corrections significantly improves reasoning factuality. Our work offers a practical approach to mitigating misinformation propagation.