Robust Answers, Fragile Logic: Probing the Decoupling Hypothesis in LLM Reasoning

TL;DR

This paper investigates whether Large Language Models' reasoning processes are genuinely reliable or just post-hoc rationalizations, revealing vulnerabilities where models maintain correct answers despite inconsistent reasoning under input perturbations.

Contribution

The paper introduces MATCHA, a novel framework for probing LLM reasoning by isolating the reasoning phase conditioned on answers, exposing the fragility of current CoT prompting methods.

Findings

LLMs often produce inconsistent reasoning when answers are correct.

Multi-step and commonsense tasks are more vulnerable to decoupling.

Adversarial examples transfer to black-box models, exposing robustness gaps.

Abstract

While Chain-of-Thought (CoT) prompting has become a cornerstone for complex reasoning in Large Language Models (LLMs), the faithfulness of the generated reasoning remains an open question. We investigate the Decoupling Hypothesis: that correct answers often mask fragile, post-hoc rationalizations that are not causally tied to the model's prediction. To systematically verify this, we introduce MATCHA, a novel Answer-Conditioned Probing framework. Unlike standard evaluations that focus on final output accuracy, MATCHA isolates the reasoning phase by conditioning generation on the model's predicted answer, allowing us to stress-test the stability of the rationale itself. Our experiments reveal a critical vulnerability: under imperceptible input perturbations, LLMs frequently maintain the correct answer while generating inconsistent or nonsensical reasoning - effectively being ``Right for the Wrong Reasons''. Using LLM judges to quantify this robustness gap, we find that multi-step and commonsense tasks are significantly more susceptible to this decoupling than logical tasks. Furthermore, we demonstrate that adversarial examples generated by MATCHA transfer non-trivially to black-box models. Our findings expose the illusion of CoT robustness and underscore the need for future architectures that enforce genuine answer-reasoning consistency rather than mere surface-level accuracy.