Self-Supervised Learning with Kernel Dependence Maximization

TL;DR

This paper introduces SSL-HSIC, a self-supervised learning framework that maximizes statistical dependence between transformed image representations, providing a new understanding of mutual information bounds and achieving state-of-the-art results.

Contribution

It proposes SSL-HSIC, a novel dependence-maximization approach for self-supervised learning, offering theoretical insights and practical improvements over existing methods like InfoNCE and BYOL.

Findings

SSL-HSIC matches state-of-the-art performance on ImageNet

It enables direct dependence optimization with linear time complexity

Effective across various vision tasks including segmentation and depth estimation

Abstract

We approach self-supervised learning of image representations from a statistical dependence perspective, proposing Self-Supervised Learning with the Hilbert-Schmidt Independence Criterion (SSL-HSIC). SSL-HSIC maximizes dependence between representations of transformations of an image and the image identity, while minimizing the kernelized variance of those representations. This framework yields a new understanding of InfoNCE, a variational lower bound on the mutual information (MI) between different transformations. While the MI itself is known to have pathologies which can result in learning meaningless representations, its bound is much better behaved: we show that it implicitly approximates SSL-HSIC (with a slightly different regularizer). Our approach also gives us insight into BYOL, a negative-free SSL method, since SSL-HSIC similarly learns local neighborhoods of samples. SSL-HSIC allows us to directly optimize statistical dependence in time linear in the batch size, without restrictive data assumptions or indirect mutual information estimators. Trained with or without a target network, SSL-HSIC matches the current state-of-the-art for standard linear evaluation on ImageNet, semi-supervised learning and transfer to other classification and vision tasks such as semantic segmentation, depth estimation and object recognition. Code is available at https://github.com/deepmind/ssl_hsic .