URLOST: Unsupervised Representation Learning without Stationarity or Topology

TL;DR

URLOST introduces a novel unsupervised learning framework that effectively learns from high-dimensional data without relying on domain-specific stationarity or topology, inspired by biological systems.

Contribution

The paper presents URLOST, a new model combining self-organizing layers, spectral clustering, and masked autoencoders to learn representations across diverse data modalities without prior topological knowledge.

Findings

Outperforms state-of-the-art methods like SimCLR and MAE on various data modalities.

Successfully learns meaningful representations from biological vision data, neural recordings, and gene expressions.

Sets a new benchmark in unsupervised learning for high-dimensional, non-stationary, and irregular data.

Abstract

Unsupervised representation learning has seen tremendous progress. However, it is constrained by its reliance on domain specific stationarity and topology, a limitation not found in biological intelligence systems. For instance, unlike computer vision, human vision can process visual signals sampled from highly irregular and non-stationary sensors. We introduce a novel framework that learns from high-dimensional data without prior knowledge of stationarity and topology. Our model, abbreviated as URLOST, combines a learnable self-organizing layer, spectral clustering, and a masked autoencoder (MAE). We evaluate its effectiveness on three diverse data modalities including simulated biological vision data, neural recordings from the primary visual cortex, and gene expressions. Compared to state-of-the-art unsupervised learning methods like SimCLR and MAE, our model excels at learning meaningful representations across diverse modalities without knowing their stationarity or topology. It also outperforms other methods that are not dependent on these factors, setting a new benchmark in the field. We position this work as a step toward unsupervised learning methods capable of generalizing across diverse high-dimensional data modalities.