Evolution of SAE Features Across Layers in LLMs

TL;DR

This paper investigates how features in transformer-based language models evolve across layers by analyzing their statistical relationships, visualizing feature communities, and identifying feature transformations and specializations.

Contribution

It introduces a method to analyze feature evolution across layers, including visualization tools and community detection, revealing feature passing, boolean combinations, and specialization.

Findings

Features are often passed through layers unchanged.

Some features are quasi-boolean combinations of previous features.

Features become more specialized in later layers.

Abstract

Sparse Autoencoders for transformer-based language models are typically defined independently per layer. In this work we analyze statistical relationships between features in adjacent layers to understand how features evolve through a forward pass. We provide a graph visualization interface for features and their most similar next-layer neighbors (https://stefanhex.com/spar-2024/feature-browser/), and build communities of related features across layers. We find that a considerable amount of features are passed through from a previous layer, some features can be expressed as quasi-boolean combinations of previous features, and some features become more specialized in later layers.