Learning Sparse Visual Representations via Spatial-Semantic Factorization

TL;DR

STELLAR introduces a novel framework that factorizes visual features into semantic concepts and spatial distributions, enabling sparse tokens to achieve both high-quality reconstruction and strong semantic understanding in self-supervised learning.

Contribution

It proposes a new factorization method that disentangles semantics and spatial information, bridging the gap between discriminative and generative SSL methods.

Findings

As few as 16 sparse tokens support high-quality reconstruction (FID 2.60).

Achieves 79.10% ImageNet accuracy with sparse tokens, matching dense backbone performance.

Demonstrates effective semantic and spatial feature disentanglement in SSL.

Abstract

Self-supervised learning (SSL) faces a fundamental conflict between semantic understanding and image reconstruction. High-level semantic SSL (e.g., DINO) relies on global tokens that are forced to be location-invariant for augmentation alignment, a process that inherently discards the spatial coordinates required for reconstruction. Conversely, generative SSL (e.g., MAE) preserves dense feature grids for reconstruction but fails to produce high-level abstractions. We introduce STELLAR, a framework that resolves this tension by factorizing visual features into a low-rank product of semantic concepts and their spatial distributions. This disentanglement allows us to perform DINO-style augmentation alignment on the semantic tokens while maintaining the precise spatial mapping in the localization matrix necessary for pixel-level reconstruction. We demonstrate that as few as 16 sparse tokens under this factorized form are sufficient to simultaneously support high-quality reconstruction (2.60 FID) and match the semantic performance of dense backbones (79.10% ImageNet accuracy). Our results highlight STELLAR as a versatile sparse representation that bridges the gap between discriminative and generative vision by strategically separating semantic identity from spatial geometry. Code available at https://aka.ms/stellar.