Can LLMs Reconcile Knowledge Conflicts in Counterfactual Reasoning

TL;DR

This paper investigates the ability of large language models to perform counterfactual reasoning by integrating parametric knowledge with new information, revealing significant limitations in their current capabilities.

Contribution

The study demonstrates that LLMs struggle with counterfactual reasoning and that simple finetuning often degrades their existing knowledge, highlighting key limitations.

Findings

LLMs often rely solely on parametric knowledge in counterfactual scenarios.

Simple finetuning can impair LLMs' existing knowledge.

LLMs show limited ability to adapt parametric knowledge to new contexts.

Abstract

Large Language Models have been shown to contain extensive world knowledge in their parameters, enabling impressive performance on many knowledge intensive tasks. However, when deployed in novel settings, LLMs often encounter situations where they must integrate parametric knowledge with new or unfamiliar information. In this work, we explore whether LLMs can combine knowledge in-context with their parametric knowledge through the lens of counterfactual reasoning. Through synthetic and real experiments in multi-hop reasoning problems, we show that LLMs generally struggle with counterfactual reasoning, often resorting to exclusively using their parametric knowledge. Moreover, we show that simple post-hoc finetuning can struggle to instill counterfactual reasoning ability -- often leading to degradation in stored parametric knowledge. Ultimately, our work reveals important limitations of current LLM's abilities to re-purpose parametric knowledge in novel settings.