Tracr-Injection: Distilling Algorithms into Pre-trained Language Models

TL;DR

This paper introduces tracr-injection, a novel method to embed RASP algorithms into pre-trained language models, enhancing interpretability and out-of-distribution performance by creating a symbolic subspace within the model.

Contribution

The paper presents tracr-injection, a new technique for distilling RASP algorithms into language models, bridging the gap between theoretical symbolic capabilities and practical learnability.

Findings

Injected algorithms form an interpretable subspace in the residual stream

The method improves out-of-distribution performance

Creates a symbolic mechanism within the model

Abstract

Motivated by the surge of large language models, there has been a push to formally characterize the symbolic abilities intrinsic to the transformer architecture. A programming language, called RASP, has been proposed, which can be directly compiled into transformer weights to implement these algorithms. However, the tasks that can be implemented in RASP are often uncommon to learn from natural unsupervised data, showing a mismatch between theoretical capabilities of the transformer architecture, and the practical learnability of these capabilities from unsupervised data. We propose tracr-injection, a method that allows us to distill algorithms written in RASP directly into a pre-trained language model. We showcase our method by injecting 3 different algorithms into a language model. We show how our method creates an interpretable subspace within the model's residual stream, which can be decoded into the variables present in the code of the RASP algorithm. Additionally, we found that the proposed method can improve out-of-distribution performance compared to our baseline, indicating that indeed a more symbolic mechanism is taking place in the inner workings of the model. We release the code used to run our experiments.