Self-Supervised Learning with Gaussian Processes

TL;DR

This paper introduces GPSSL, a novel self-supervised learning method using Gaussian processes to improve representation quality, uncertainty quantification, and out-of-sample prediction, outperforming traditional SSL methods.

Contribution

The paper proposes Gaussian process-based SSL (GPSSL), which imposes GP priors on representations, providing uncertainty estimates and a natural way to enforce smoothness without explicit positive pairs.

Findings

GPSSL outperforms traditional SSL methods in accuracy.

GPSSL provides meaningful uncertainty quantification.

GPSSL improves out-of-sample prediction performance.

Abstract

Self supervised learning (SSL) is a machine learning paradigm where models learn to understand the underlying structure of data without explicit supervision from labeled samples. The acquired representations from SSL have demonstrated useful for many downstream tasks including clustering, and linear classification, etc. To ensure smoothness of the representation space, most SSL methods rely on the ability to generate pairs of observations that are similar to a given instance. However, generating these pairs may be challenging for many types of data. Moreover, these methods lack consideration of uncertainty quantification and can perform poorly in out-of-sample prediction settings. To address these limitations, we propose Gaussian process self supervised learning (GPSSL), a novel approach that utilizes Gaussian processes (GP) models on representation learning. GP priors are imposed on the representations, and we obtain a generalized Bayesian posterior minimizing a loss function that encourages informative representations. The covariance function inherent in GPs naturally pulls representations of similar units together, serving as an alternative to using explicitly defined positive samples. We show that GPSSL is closely related to both kernel PCA and VICReg, a popular neural network-based SSL method, but unlike both allows for posterior uncertainties that can be propagated to downstream tasks. Experiments on various datasets, considering classification and regression tasks, demonstrate that GPSSL outperforms traditional methods in terms of accuracy, uncertainty quantification, and error control.