Sparsity and Superposition in Mixture of Experts

TL;DR

This paper investigates how sparsity and superposition affect the interpretability and specialization of mixture of experts models, revealing that increased sparsity leads to more interpretable, monosemantic expert representations without loss of performance.

Contribution

It introduces new metrics for superposition, redefines expert specialization based on feature coherence, and demonstrates that network sparsity enhances interpretability in MoE models.

Findings

Greater network sparsity correlates with increased monosemanticity.

Models with higher sparsity organize experts around coherent features.

Sparsity enables interpretability without performance loss.

Abstract

Mixture of Experts (MoE) models have become central to scaling large language models, yet their mechanistic differences from dense networks remain poorly understood. Previous work has explored how dense models use \textit{superposition} to represent more features than dimensions, and how superposition is a function of feature sparsity and feature importance. MoE models cannot be explained mechanistically through the same lens. We find that neither feature sparsity nor feature importance cause discontinuous phase changes, and that network sparsity (the ratio of active to total experts) better characterizes MoEs. We develop new metrics for measuring superposition across experts. Our findings demonstrate that models with greater network sparsity exhibit greater \emph{monosemanticity}. We propose a new definition of expert specialization based on monosemantic feature representation rather than load balancing, showing that experts naturally organize around coherent feature combinations when initialized appropriately. These results suggest that network sparsity in MoEs may enable more interpretable models without sacrificing performance, challenging the common assumption that interpretability and capability are fundamentally at odds.