Tracr: Compiled Transformers as a Laboratory for Interpretability

TL;DR

Tracr introduces a method to compile human-readable programs into transformer models, creating structures that facilitate interpretability experiments and ground-truth evaluation of interpretability methods.

Contribution

The paper presents Tracr, a compiler that generates transformer models with known, interpretable structures for research and evaluation purposes.

Findings

Tracr-compiled models accurately implement the specified programs.

The known structure enables targeted interpretability experiments.

Models successfully perform tasks like token frequency counting, sorting, and parenthesis checking.

Abstract

We show how to "compile" human-readable programs into standard decoder-only transformer models. Our compiler, Tracr, generates models with known structure. This structure can be used to design experiments. For example, we use it to study "superposition" in transformers that execute multi-step algorithms. Additionally, the known structure of Tracr-compiled models can serve as ground-truth for evaluating interpretability methods. Commonly, because the "programs" learned by transformers are unknown it is unclear whether an interpretation succeeded. We demonstrate our approach by implementing and examining programs including computing token frequencies, sorting, and parenthesis checking. We provide an open-source implementation of Tracr at https://github.com/google-deepmind/tracr.