From Atoms to Trees: Building a Structured Feature Forest with Hierarchical Sparse Autoencoders

TL;DR

This paper introduces Hierarchical Sparse Autoencoders (HSAE), a novel method for uncovering hierarchical, semantically meaningful structures in large language models, enhancing interpretability without sacrificing reconstruction quality.

Contribution

HSAE jointly learns feature hierarchies and parent-child relationships, using structural constraints and perturbations, advancing the analysis of LLM internal representations.

Findings

HSAE reliably recovers semantic hierarchies across models and layers.

HSAE maintains high reconstruction fidelity and interpretability.

Qualitative and quantitative evaluations validate HSAE's effectiveness.

Abstract

Sparse autoencoders (SAEs) have proven effective for extracting monosemantic features from large language models (LLMs), yet these features are typically identified in isolation. However, broad evidence suggests that LLMs capture the intrinsic structure of natural language, where the phenomenon of "feature splitting" in particular indicates that such structure is hierarchical. To capture this, we propose the Hierarchical Sparse Autoencoder (HSAE), which jointly learns a series of SAEs and the parent-child relationships between their features. HSAE strengthens the alignment between parent and child features through two novel mechanisms: a structural constraint loss and a random feature perturbation mechanism. Extensive experiments across various LLMs and layers demonstrate that HSAE consistently recovers semantically meaningful hierarchies, supported by both qualitative case studies and rigorous quantitative metrics. At the same time, HSAE preserves the reconstruction fidelity and interpretability of standard SAEs across different dictionary sizes. Our work provides a powerful, scalable tool for discovering and analyzing the multi-scale conceptual structures embedded in LLM representations.