Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures

TL;DR

This paper investigates whether intermediate structures in schema-guided reasoning pipelines causally influence LLM outputs, revealing that such structures often act as influential context rather than stable causal factors, with implications for model interpretability.

Contribution

Introduces a causal evaluation protocol to test the causal influence of intermediate structures on LLM outputs, demonstrating their fragility and limited causal role.

Findings

Models are self-consistent with their structures but often do not update after interventions.

Fragility of faithfulness increases when external tools derive final decisions.

Intermediate structures act more as influential context than causal mediators.

Abstract

Schema-guided reasoning pipelines ask LLMs to produce explicit intermediate structures -- rubrics, checklists, verification queries -- before committing to a final decision. But do these structures causally determine the output, or merely accompany it? We introduce a causal evaluation protocol that makes this directly measurable: by selecting tasks where a deterministic function maps intermediate structures to decisions, every controlled edit implies a unique correct output. Across eight models and three benchmarks, models appear self-consistent with their own intermediate structures but fail to update predictions after intervention in up to 60% of cases -- revealing that apparent faithfulness is fragile once the intermediate structure changes. When derivation of the final decision from the structure is delegated to an external tool, this fragility largely disappears; however, prompts which ask to prioritize the intermediate structure over the original input do not materially close the gap. Overall, intermediate structures in schema-guided pipelines function as influential context rather than stable causal mediators.