KisMATH: Do LLMs Have Knowledge of Implicit Structures in Mathematical Reasoning?

TL;DR

This paper introduces CCGraphs, a new method to analyze how large language models perform mathematical reasoning by modeling causal dependencies in their reasoning traces, revealing that models internalize similar structures.

Contribution

The paper presents CCGraphs and the KisMATH dataset, providing empirical evidence that LLMs recognize and utilize causal structures in mathematical reasoning.

Findings

Reasoning nodes are causal contributors to answers.

LLMs emphasize reasoning paths similar to CCGraphs.

CCGraphs enable controlled interventions in reasoning analysis.

Abstract

Chain-of-thought (CoT) traces have been shown to improve performance of large language models on a plethora of reasoning tasks, yet there is no consensus on the mechanism by which this boost is achieved. To shed more light on this, we introduce Causal CoT Graphs (CCGraphs), which are directed acyclic graphs automatically extracted from reasoning traces that model fine-grained causal dependencies in language-model outputs. A collection of 1671 mathematical reasoning problems from MATH500, GSM8K, and AIME, together with their associated CCGraphs, has been compiled into our dataset -- KisMATH. Our detailed empirical analysis with 15 open-weight LLMs shows that (i) reasoning nodes in the CCGraphs are causal contributors to the final answer, which we argue is constitutive of reasoning; and (ii) LLMs emphasize the reasoning paths captured by the CCGraphs, indicating that the models internally realize structures similar to our graphs. KisMATH enables controlled, graph-aligned interventions and opens avenues for further investigation into the role of CoT in LLM reasoning.