Mechanistic interpretability of large language models with applications to the financial services industry

TL;DR

This paper applies mechanistic interpretability to large language models like GPT-2 to understand their internal decision processes, especially for financial compliance tasks, using attention analysis and causal interventions.

Contribution

It pioneers the use of mechanistic interpretability techniques to analyze LLMs in financial applications, identifying key attention heads involved in specific tasks.

Findings

Attention heads 10.2, 10.7, 11.3 are positively involved in task completion.

Heads 9.6 and 10.6 negatively influence the task.

Activation patching helps localize task-specific components.

Abstract

Large Language Models such as GPTs (Generative Pre-trained Transformers) exhibit remarkable capabilities across a broad spectrum of applications. Nevertheless, due to their intrinsic complexity, these models present substantial challenges in interpreting their internal decision-making processes. This lack of transparency poses critical challenges when it comes to their adaptation by financial institutions, where concerns and accountability regarding bias, fairness, and reliability are of paramount importance. Mechanistic interpretability aims at reverse engineering complex AI models such as transformers. In this paper, we are pioneering the use of mechanistic interpretability to shed some light on the inner workings of large language models for use in financial services applications. We offer several examples of how algorithmic tasks can be designed for compliance monitoring purposes. In particular, we investigate GPT-2 Small's attention pattern when prompted to identify potential violation of Fair Lending laws. Using direct logit attribution, we study the contributions of each layer and its corresponding attention heads to the logit difference in the residual stream. Finally, we design clean and corrupted prompts and use activation patching as a causal intervention method to localize our task completion components further. We observe that the (positive) heads $10.2$ (head $2$, layer $10$), $10.7$, and $11.3$, as well as the (negative) heads $9.6$ and $10.6$ play a significant role in the task completion.