The Persian Rug: solving toy models of superposition using large-scale symmetries

TL;DR

This paper provides a detailed analysis of a sparse autoencoder's learned algorithm in high-dimensional, permutation-symmetric data, revealing large-scale symmetry effects and introducing a new interpretability technique, with implications for neural network understanding.

Contribution

It offers an explicit analytical description of the autoencoder's behavior under symmetry, demonstrating near-optimal performance and linking learned weights to fractal structures.

Findings

Large models learn symmetry-sensitive algorithms based on weight statistics.

The loss function becomes analytically tractable at high sparsity.

Artificial weights exhibit fractal structures resembling Persian rugs.

Abstract

We present a complete mechanistic description of the algorithm learned by a minimal non-linear sparse data autoencoder in the limit of large input dimension. The model, originally presented in arXiv:2209.10652, compresses sparse data vectors through a linear layer and decompresses using another linear layer followed by a ReLU activation. We notice that when the data is permutation symmetric (no input feature is privileged) large models reliably learn an algorithm that is sensitive to individual weights only through their large-scale statistics. For these models, the loss function becomes analytically tractable. Using this understanding, we give the explicit scalings of the loss at high sparsity, and show that the model is near-optimal among recently proposed architectures. In particular, changing or adding to the activation function any elementwise or filtering operation can at best improve the model's performance by a constant factor. Finally, we forward-engineer a model with the requisite symmetries and show that its loss precisely matches that of the trained models. Unlike the trained model weights, the low randomness in the artificial weights results in miraculous fractal structures resembling a Persian rug, to which the algorithm is oblivious. Our work contributes to neural network interpretability by introducing techniques for understanding the structure of autoencoders. Code to reproduce our results can be found at https://github.com/KfirD/PersianRug .