Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders

TL;DR

This paper introduces Mechanistic Topic Models (MTMs) that leverage sparse autoencoders to learn interpretable, semantically rich features, enabling deeper thematic insights and controllable text generation, outperforming traditional models on multiple datasets.

Contribution

MTMs utilize autoencoder-derived features for more expressive and interpretable topics, and introduce a new evaluation framework with LLM-based pairwise comparisons.

Findings

MTMs match or outperform baselines on coherence metrics

MTMs are preferred by the topic judge in evaluations

MTMs enable effective steering of LLM outputs

Abstract

Traditional topic models are effective at uncovering latent themes in large text collections. However, due to their reliance on bag-of-words representations, they struggle to capture semantically abstract features. While some neural variants use richer representations, they are similarly constrained by expressing topics as word lists, which limits their ability to articulate complex topics. We introduce Mechanistic Topic Models (MTMs), a class of topic models that operate on interpretable features learned by sparse autoencoders (SAEs). By defining topics over this semantically rich space, MTMs can reveal deeper conceptual themes with expressive feature descriptions. Moreover, uniquely among topic models, MTMs enable controllable text generation using topic-based steering vectors. To properly evaluate MTM topics against word-list-based approaches, we propose \textit{topic judge}, an LLM-based pairwise comparison evaluation framework. Across five datasets, MTMs match or exceed traditional and neural baselines on coherence metrics, are consistently preferred by topic judge, and enable effective steering of LLM outputs.