Interpreting Video Representations with Spatio-Temporal Sparse Autoencoders

TL;DR

This paper systematically studies Sparse Autoencoders for video representations, introducing contrastive objectives and hierarchical grouping to improve temporal coherence and interpretability, with applications in action recognition and retrieval.

Contribution

It proposes novel spatio-temporal contrastive training methods and hierarchical grouping for SAEs, enhancing temporal coherence and interpretability of video features.

Findings

Contrastive SAE features improve action classification by +3.9%.

Contrastive training concentrates predictive signals into fewer features.

Different configurations excel at reconstruction, coherence, or interpretability.

Abstract

We present the first systematic study of Sparse Autoencoders (SAEs) on video representations. Standard SAEs decompose video into interpretable, monosemantic features but destroy temporal coherence: hard TopK selection produces unstable feature assignments across frames, reducing autocorrelation by 36%. We propose spatio-temporal contrastive objectives and Matryoshka hierarchical grouping that recover and even exceed raw temporal coherence. The contrastive loss weight controls a tunable trade-off between reconstruction and temporal coherence. A systematic ablation on two backbones and two datasets shows that different configurations excel at different goals: reconstruction fidelity, temporal coherence, action discrimination, or interpretability. Contrastive SAE features improve action classification by +3.9% over raw features and text-video retrieval by up to 2.8xR@1. A cross-backbone analysis reveals that standard monosemanticity metrics contain a backbone-alignment artifact: both DINOv2 and VideoMAE produce equally monosemantic features under neutral (CLIP) similarity. Causal ablation confirms that contrastive training concentrates predictive signal into a small number of identifiable features.