Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoning

TL;DR

This paper investigates the internal reasoning mechanisms of GPT-2 using circuit analysis and self-influence functions, revealing human-interpretable multi-step reasoning paths in language models.

Contribution

It introduces a novel approach combining circuit analysis and self-influence to interpret multi-step reasoning in transformer models.

Findings

GPT-2's reasoning process can be mapped to human-interpretable circuits.

Self-influence functions highlight token importance changes during reasoning.

The methodology uncovers explicit reasoning paths in transformer models.

Abstract

Transformer-based language models have achieved significant success; however, their internal mechanisms remain largely opaque due to the complexity of non-linear interactions and high-dimensional operations. While previous studies have demonstrated that these models implicitly embed reasoning trees, humans typically employ various distinct logical reasoning mechanisms to complete the same task. It is still unclear which multi-step reasoning mechanisms are used by language models to solve such tasks. In this paper, we aim to address this question by investigating the mechanistic interpretability of language models, particularly in the context of multi-step reasoning tasks. Specifically, we employ circuit analysis and self-influence functions to evaluate the changing importance of each token throughout the reasoning process, allowing us to map the reasoning paths adopted by the model. We apply this methodology to the GPT-2 model on a prediction task (IOI) and demonstrate that the underlying circuits reveal a human-interpretable reasoning process used by the model.