Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability

TL;DR

This paper introduces Temporal Sparse Autoencoders (T-SAEs), which incorporate a contrastive loss to leverage the sequential nature of language, resulting in more coherent and interpretable semantic features without sacrificing reconstruction quality.

Contribution

The paper proposes T-SAEs that utilize temporal structure and contrastive loss to improve interpretability of language models by disentangling semantic features in an unsupervised manner.

Findings

T-SAEs recover smoother, more coherent semantic concepts.

They outperform traditional SAEs in interpretability without losing reconstruction quality.

Semantic structure emerges clearly even without explicit semantic supervision.

Abstract

Translating the internal representations and computations of models into concepts that humans can understand is a key goal of interpretability. While recent dictionary learning methods such as Sparse Autoencoders (SAEs) provide a promising route to discover human-interpretable features, they often only recover token-specific, noisy, or highly local concepts. We argue that this limitation stems from neglecting the temporal structure of language, where semantic content typically evolves smoothly over sequences. Building on this insight, we introduce Temporal Sparse Autoencoders (T-SAEs), which incorporate a novel contrastive loss encouraging consistent activations of high-level features over adjacent tokens. This simple yet powerful modification enables SAEs to disentangle semantic from syntactic features in a self-supervised manner. Across multiple datasets and models, T-SAEs recover smoother, more coherent semantic concepts without sacrificing reconstruction quality. Strikingly, they exhibit clear semantic structure despite being trained without explicit semantic signal, offering a new pathway for unsupervised interpretability in language models.