On minimal variations for unsupervised representation learning

TL;DR

This paper explores the principle of minimal variations as a unifying concept in unsupervised representation learning, aiming to improve understanding and development of self-supervised algorithms.

Contribution

It introduces the idea that minimal variations underpin many existing techniques, providing new insights and guidelines for designing self-supervised learning methods.

Findings

Minimal variations serve as a common principle across various unsupervised methods

Unveiling this principle offers practical guidelines for algorithm development

Enhances understanding of the assumptions behind representation learning techniques

Abstract

Unsupervised representation learning aims at describing raw data efficiently to solve various downstream tasks. It has been approached with many techniques, such as manifold learning, diffusion maps, or more recently self-supervised learning. Those techniques are arguably all based on the underlying assumption that target functions, associated with future downstream tasks, have low variations in densely populated regions of the input space. Unveiling minimal variations as a guiding principle behind unsupervised representation learning paves the way to better practical guidelines for self-supervised learning algorithms.