Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisions

TL;DR

This paper demonstrates that large language models can be trained to accurately describe their internal decision-making processes, enhancing interpretability and safety by enabling models to explain their own reasoning across diverse complex tasks.

Contribution

It introduces methods to fine-tune LLMs for self-interpretability, allowing them to explain their internal processes and decision weights more accurately and generally.

Findings

LLMs can accurately report their internal decision weights.

Fine-tuning improves models' ability to explain their decision-making.

Training generalizes to various complex decisions.

Abstract

We have only limited understanding of how and why large language models (LLMs) respond in the ways that they do. Their neural networks have proven challenging to interpret, and we are only beginning to tease out the function of individual neurons and circuits within them. However, another path to understanding these systems is to investigate and develop their capacity to explain their own functioning. Here, we show that i) LLMs can accurately describe quantitative features of their own internal processes during certain kinds of decision-making and ii) that it is possible to improve these capabilities through training. To do so, we fine-tuned GPT-4o and GPT-4o-mini to make decisions in a wide variety of complex contexts (e.g., choosing between condos, loans, vacations, etc.) according to randomly-generated, quantitative preferences about how to weigh different attributes (e.g., the relative importance of natural light versus quiet surroundings for condos). We demonstrate that the LLMs can accurately report these preferences (i.e., the weights that they learned to give to different attributes during decision-making). Next, we demonstrate that these LLMs can be fine-tuned to explain their decision-making even more accurately. Finally, we demonstrate that this training generalizes: It improves the ability of the models to accurately explain how they make other complex decisions, not just decisions they have been fine-tuned to make. This work is a step towards training LLMs to accurately and broadly report on their own internal processes -- a possibility that would yield substantial benefits for interpretability, control, and safety.