Non-Parametric Representation Learning with Kernels

TL;DR

This paper introduces kernel-based methods for unsupervised and self-supervised representation learning, including contrastive kernel SSL models and a kernel autoencoder, with theoretical analysis and empirical evaluation.

Contribution

It presents novel kernel-based models for representation learning, along with new theoretical results and generalisation bounds, expanding beyond neural network approaches.

Findings

Kernel SSL models outperform neural networks in small data regimes.

New representer theorems relate kernel representations to kernel matrices.

Empirical results demonstrate competitive performance of kernel methods.

Abstract

Unsupervised and self-supervised representation learning has become popular in recent years for learning useful features from unlabelled data. Representation learning has been mostly developed in the neural network literature, and other models for representation learning are surprisingly unexplored. In this work, we introduce and analyze several kernel-based representation learning approaches: Firstly, we define two kernel Self-Supervised Learning (SSL) models using contrastive loss functions and secondly, a Kernel Autoencoder (AE) model based on the idea of embedding and reconstructing data. We argue that the classical representer theorems for supervised kernel machines are not always applicable for (self-supervised) representation learning, and present new representer theorems, which show that the representations learned by our kernel models can be expressed in terms of kernel matrices. We further derive generalisation error bounds for representation learning with kernel SSL and AE, and empirically evaluate the performance of these methods in both small data regimes as well as in comparison with neural network based models.