Automatically Finding Rule-Based Neurons in OthelloGPT

TL;DR

This paper introduces an automated decision tree-based method to interpret neurons in OthelloGPT, revealing rule-based patterns and verifying their causal role in move prediction.

Contribution

It presents a novel approach to automatically extract human-readable rule-based neuron interpretations in a transformer model for Othello, enabling better understanding of its internal logic.

Findings

Approximately half of the neurons are described by rule-based decision trees with high accuracy.

Targeted neuron ablations significantly impair move prediction along identified patterns.

Provides a Python tool for mapping game behaviors to neurons for future interpretability research.

Abstract

OthelloGPT, a transformer trained to predict valid moves in Othello, provides an ideal testbed for interpretability research. The model is complex enough to exhibit rich computational patterns, yet grounded in rule-based game logic that enables meaningful reverse-engineering. We present an automated approach based on decision trees to identify and interpret MLP neurons that encode rule-based game logic. Our method trains regression decision trees to map board states to neuron activations, then extracts decision paths where neurons are highly active to convert them into human-readable logical forms. These descriptions reveal highly interpretable patterns; for instance, neurons that specifically detect when diagonal moves become legal. Our findings suggest that roughly half of the neurons in layer 5 can be accurately described by compact, rule-based decision trees ($R^2 > 0.7$ for 913 of 2,048 neurons), while the remainder likely participate in more distributed or non-rule-based computations. We verify the causal relevance of patterns identified by our decision trees through targeted interventions. For a specific square, for specific game patterns, we ablate neurons corresponding to those patterns and find an approximately 5-10 fold stronger degradation in the model's ability to predict legal moves along those patterns compared to control patterns. To facilitate future work, we provide a Python tool that maps rule-based game behaviors to their implementing neurons, serving as a resource for researchers to test whether their interpretability methods recover meaningful computational structures.