Diagnosing and Mitigating Sycophancy and Skepticism in LLM Causal Judgment

TL;DR

This paper introduces CAUSALT3, a benchmark for causal reasoning in large language models, identifies key pathologies affecting reasoning under social pressure, and proposes RCA as an inference-time mitigation to improve trustworthiness.

Contribution

It presents a new benchmark and evaluation framework for causal reasoning, uncovers specific reasoning failures, and offers RCA as a novel inference-time control method to mitigate these issues.

Findings

CAUSALT3 enables detailed analysis of LLM causal reasoning performance.

Identifies three key pathologies: Skepticism Trap, Sycophancy Trap, and Scaling Paradox.

RCA effectively reduces sycophantic acceptance to near zero without retraining.

Abstract

Large language models increasingly fail in a way that scalar accuracy cannot diagnose: they produce a sound reasoning trace and then abandon it under social pressure or an authoritative hint. We argue that this is a control failure, not a knowledge failure, and that it requires an evaluation surface richer than a single accuracy number. We introduce CAUSALT3, a 454 instance expert curated benchmark for causal reasoning across all three rungs of Pearl's ladder, and a three axis evaluation that decomposes performance into Utility (sensitivity to valid causal claims), Safety (specificity against invalid ones), and Wise Refusal (calibrated abstention on genuinely underdetermined items). On this surface we document three reproducible pathologies: a Skepticism Trap at L1 where capable models over refuse sound links, a Sycophancy Trap at L2 where confident user pressure flips correct answers, and a Scaling Paradox at L3 where a frontier model underperforms an older one on counterfactual Safety by 55 points. To mitigate these failures without retraining, we propose Regulated Causal Anchoring (RCA), an inference time process verifier that audits trace output consistency under a PID style feedback loop and abstains rather than ratifying a detected mismatch. Across CAUSALT3 and a supporting CAP-GSM8K stress test, RCA reduces sycophantic acceptance to near zero while preserving valid hint acceptance, recasting trustworthy reasoning as a question of inference time control rather than scale.