Evaluating Synthetic Activations composed of SAE Latents in GPT-2

TL;DR

This paper investigates the similarity between real and synthetic activations in GPT-2, showing that SAE-based synthetic activations can closely mimic real ones when considering their geometric and statistical properties.

Contribution

The study introduces a method to generate synthetic activations using SAE latents and compares their properties to real activations in GPT-2, revealing structural differences.

Findings

Synthetic activations resemble real activations when controlling for sparsity and cosine similarity.

Real activations exhibit more pronounced activation plateaus than synthetic ones.

SAE latents have significant geometric and statistical properties influencing activations.

Abstract

Sparse Auto-Encoders (SAEs) are commonly employed in mechanistic interpretability to decompose the residual stream into monosemantic SAE latents. Recent work demonstrates that perturbing a model's activations at an early layer results in a step-function-like change in the model's final layer activations. Furthermore, the model's sensitivity to this perturbation differs between model-generated (real) activations and random activations. In our study, we assess model sensitivity in order to compare real activations to synthetic activations composed of SAE latents. Our findings indicate that synthetic activations closely resemble real activations when we control for the sparsity and cosine similarity of the constituent SAE latents. This suggests that real activations cannot be explained by a simple "bag of SAE latents" lacking internal structure, and instead suggests that SAE latents possess significant geometric and statistical properties. Notably, we observe that our synthetic activations exhibit less pronounced activation plateaus compared to those typically surrounding real activations.