Minimalistic Unsupervised Learning with the Sparse Manifold Transform

TL;DR

This paper introduces a minimalistic, interpretable unsupervised learning method using the sparse manifold transform, achieving near state-of-the-art results without complex engineering or data augmentation.

Contribution

The paper presents a simple, deterministic sparse manifold transform approach that unifies several learning principles and achieves competitive accuracy on standard datasets.

Findings

Achieves 99.3% KNN top-1 accuracy on MNIST

Reaches 81.1% on CIFAR-10 and 53.2% on CIFAR-100 without augmentation

Close to SOTA performance with a transparent, white-box method

Abstract

We describe a minimalistic and interpretable method for unsupervised learning, without resorting to data augmentation, hyperparameter tuning, or other engineering designs, that achieves performance close to the SOTA SSL methods. Our approach leverages the sparse manifold transform, which unifies sparse coding, manifold learning, and slow feature analysis. With a one-layer deterministic sparse manifold transform, one can achieve 99.3% KNN top-1 accuracy on MNIST, 81.1% KNN top-1 accuracy on CIFAR-10 and 53.2% on CIFAR-100. With a simple gray-scale augmentation, the model gets 83.2% KNN top-1 accuracy on CIFAR-10 and 57% on CIFAR-100. These results significantly close the gap between simplistic "white-box" methods and the SOTA methods. Additionally, we provide visualization to explain how an unsupervised representation transform is formed. The proposed method is closely connected to latent-embedding self-supervised methods and can be treated as the simplest form of VICReg. Though there remains a small performance gap between our simple constructive model and SOTA methods, the evidence points to this as a promising direction for achieving a principled and white-box approach to unsupervised learning.