CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching

TL;DR

This paper introduces CausalFlip, a benchmark for evaluating and improving large language models' ability to perform true causal reasoning beyond semantic pattern matching, using specially designed questions and evaluation methods.

Contribution

The paper proposes CausalFlip, a novel causal reasoning benchmark with adversarial question pairs and noisy-prefix evaluation, and assesses different training paradigms to enhance causal reasoning in LLMs.

Findings

Explicit Chain-of-Thought can be misled by semantic correlations.

Internalized causal reasoning improves causal grounding.

Models trained with internalized reasoning outperform answer-only and explicit CoT methods.

Abstract

As large language models (LLMs) witness increasing deployment in complex, high-stakes decision-making scenarios, it becomes imperative to ground their reasoning in causality rather than spurious correlations. However, strong performance on traditional reasoning benchmarks does not guarantee true causal reasoning ability of LLMs, as high accuracy may still arise from memorizing semantic patterns instead of analyzing the underlying true causal structures. To bridge this critical gap, we propose a new causal reasoning benchmark, CausalFlip, designed to encourage the development of new LLM paradigm or training algorithms that ground LLM reasoning in causality rather than semantic correlation. CausalFlip consists of causal judgment questions built over event triples that could form different confounder, chain, and collider relations. Based on this, for each event triple, we construct pairs of semantically similar questions that reuse the same events but yield opposite causal answers, where models that rely heavily on semantic matching are systematically driven toward incorrect predictions. To further probe models' reliance on semantic patterns, we introduce a noisy-prefix evaluation that prepends causally irrelevant text before intermediate causal reasoning steps without altering the underlying causal relations or the logic of the reasoning process. We evaluate LLMs under multiple training paradigms, including answer-only training, explicit Chain-of-Thought (CoT) supervision, and a proposed internalized causal reasoning approach that aims to mitigate explicit reliance on correlation in the reasoning process. Our results show that explicit CoT can still be misled by spurious semantic correlations, where internalizing reasoning steps yields substantially improved causal grounding, suggesting that it is promising to better elicit the latent causal reasoning capabilities of base LLMs.