Bilinear MLPs enable weight-based mechanistic interpretability

TL;DR

This paper introduces bilinear MLPs, a type of neural network layer without element-wise nonlinearities, enabling direct interpretability of weights and revealing insights into model computations across various tasks.

Contribution

The paper demonstrates that bilinear MLPs allow for weight-based interpretability, providing a fully linear framework to analyze and understand neural network weights.

Findings

Bilinear MLPs achieve competitive performance without element-wise nonlinearities.

Spectral analysis reveals low-rank interpretable structures in weights.

Weight-based interpretability enables analysis of adversarial examples and overfitting.

Abstract

A mechanistic understanding of how MLPs do computation in deep neural networks remains elusive. Current interpretability work can extract features from hidden activations over an input dataset but generally cannot explain how MLP weights construct features. One challenge is that element-wise nonlinearities introduce higher-order interactions and make it difficult to trace computations through the MLP layer. In this paper, we analyze bilinear MLPs, a type of Gated Linear Unit (GLU) without any element-wise nonlinearity that nevertheless achieves competitive performance. Bilinear MLPs can be fully expressed in terms of linear operations using a third-order tensor, allowing flexible analysis of the weights. Analyzing the spectra of bilinear MLP weights using eigendecomposition reveals interpretable low-rank structure across toy tasks, image classification, and language modeling. We use this understanding to craft adversarial examples, uncover overfitting, and identify small language model circuits directly from the weights alone. Our results demonstrate that bilinear layers serve as an interpretable drop-in replacement for current activation functions and that weight-based interpretability is viable for understanding deep-learning models.