Flying Pigs, FaR and Beyond: Evaluating LLM Reasoning in Counterfactual Worlds

TL;DR

This paper evaluates how well large language models reason in counterfactual scenarios where their knowledge conflicts with the context, introduces a new benchmark, and proposes a metacognitive intervention to improve reasoning accuracy.

Contribution

It introduces CounterLogic, a benchmark for testing LLM reasoning in counterfactual worlds, and proposes Flag & Reason (FaR), a simple intervention that enhances model robustness.

Findings

LLMs' accuracy drops by 14% in counterfactual scenarios.

Flag & Reason reduces the performance gap to 7%.

Metacognitive prompting improves overall reasoning accuracy.

Abstract

A fundamental challenge in reasoning is navigating hypothetical, counterfactual worlds where logic may conflict with ingrained knowledge. We investigate this frontier for Large Language Models (LLMs) by asking: Can LLMs reason logically when the context contradicts their parametric knowledge? To facilitate a systematic analysis, we first introduce CounterLogic, a benchmark specifically designed to disentangle logical validity from knowledge alignment. Evaluation of 11 LLMs across six diverse reasoning datasets reveals a consistent failure: model accuracy plummets by an average of 14% in counterfactual scenarios compared to knowledge-aligned ones. We hypothesize that this gap stems not from a flaw in logical processing, but from an inability to manage the cognitive conflict between context and knowledge. Inspired by human metacognition, we propose a simple yet powerful intervention: Flag & Reason (FaR), where models are first prompted to flag potential knowledge conflicts before they reason. This metacognitive step is highly effective, narrowing the performance gap to just 7% and increasing overall accuracy by 4%. Our findings diagnose and study a critical limitation in modern LLMs' reasoning and demonstrate how metacognitive awareness can make them more robust and reliable thinkers.