Self-Supervised Learning Using Nonlinear Dependence

TL;DR

This paper introduces CDSSL, a self-supervised learning framework that captures both linear and nonlinear dependencies in data using HSIC, leading to improved representations especially in complex high-dimensional visual datasets.

Contribution

The paper proposes a novel SSL method that unifies linear correlation and nonlinear dependence modeling through HSIC, extending existing approaches.

Findings

Enhanced representation quality on multiple benchmarks

Effective capture of nonlinear dependencies in high-dimensional data

Outperforms existing SSL methods in experiments

Abstract

Self-supervised learning has gained significant attention in contemporary applications, particularly due to the scarcity of labeled data. While existing SSL methodologies primarily address feature variance and linear correlations, they often neglect the intricate relations between samples and the nonlinear dependencies inherent in complex data--especially prevalent in high-dimensional visual data. In this paper, we introduce Correlation-Dependence Self-Supervised Learning (CDSSL), a novel framework that unifies and extends existing SSL paradigms by integrating both linear correlations and nonlinear dependencies, encapsulating sample-wise and feature-wise interactions. Our approach incorporates the Hilbert-Schmidt Independence Criterion (HSIC) to robustly capture nonlinear dependencies within a Reproducing Kernel Hilbert Space, enriching representation learning. Experimental evaluations on diverse benchmarks demonstrate the efficacy of CDSSL in improving representation quality.