Tracing the Thought of a Grandmaster-level Chess-Playing Transformer

TL;DR

This paper introduces a novel sparse decomposition framework to interpret the internal computation of Leela Chess Zero, revealing its tactical reasoning and behavior in detail.

Contribution

It is the first work to decompose both MLP and attention modules in a transformer for interpretability, providing insights into superhuman chess reasoning.

Findings

LC0 exhibits interpretable tactical pathways.

Three quantitative metrics show parallel reasoning behavior.

Code for the framework is publicly available.

Abstract

While modern transformer neural networks achieve grandmaster-level performance in chess and other reasoning tasks, their internal computation process remains largely opaque. Focusing on Leela Chess Zero (LC0), we introduce a sparse decomposition framework to interpret its internal computation by decomposing its MLP and attention modules with sparse replacement layers, which capture the primary computation process of LC0. We conduct a detailed case study showing that these pathways expose rich, interpretable tactical considerations that are empirically verifiable. We further introduce three quantitative metrics and show that LC0 exhibits parallel reasoning behavior consistent with the inductive bias of its policy head architecture. To the best of our knowledge, this is the first work to decompose the internal computation of a transformer on both MLP and attention modules for interpretability. Combining sparse replacement layers and causal interventions in LC0 provides a comprehensive understanding of advanced tactical reasoning, offering critical insights into the underlying mechanisms of superhuman systems. Our code is available at https://github.com/JacklE0niden/Leela-SAEs.