Beyond Redundancy: Diverse and Specialized Multi-Expert Sparse Autoencoder

TL;DR

This paper introduces novel methods for multi-expert sparse autoencoders that improve feature specialization and diversity, significantly reducing redundancy and reconstruction error, thus enhancing interpretability of large language models efficiently.

Contribution

It proposes two innovations—Multiple Expert Activation and Feature Scaling—to address expert specialization issues in MoE-SAE, improving interpretability and efficiency.

Findings

24% lower reconstruction error

99% reduction in feature redundancy

Enhanced interpretability of LLMs

Abstract

Sparse autoencoders (SAEs) have emerged as a powerful tool for interpreting large language models (LLMs) by decomposing token activations into combinations of human-understandable features. While SAEs provide crucial insights into LLM explanations, their practical adoption faces a fundamental challenge: better interpretability demands that SAEs' hidden layers have high dimensionality to satisfy sparsity constraints, resulting in prohibitive training and inference costs. Recent Mixture of Experts (MoE) approaches attempt to address this by partitioning SAEs into narrower expert networks with gated activation, thereby reducing computation. In a well-designed MoE, each expert should focus on learning a distinct set of features. However, we identify a \textit{critical limitation} in MoE-SAE: Experts often fail to specialize, which means they frequently learn overlapping or identical features. To deal with it, we propose two key innovations: (1) Multiple Expert Activation that simultaneously engages semantically weighted expert subsets to encourage specialization, and (2) Feature Scaling that enhances diversity through adaptive high-frequency scaling. Experiments demonstrate a 24\% lower reconstruction error and a 99\% reduction in feature redundancy compared to existing MoE-SAE methods. This work bridges the interpretability-efficiency gap in LLM analysis, allowing transparent model inspection without compromising computational feasibility.