Radial-VCReg: More Informative Representation Learning Through Radial Gaussianization

TL;DR

Radial-VCReg enhances self-supervised representation learning by applying radial Gaussianization, aligning feature norms with the Chi distribution, and outperforming previous methods in promoting diverse, informative features.

Contribution

It introduces Radial-VCReg, a novel regularization technique that extends VCReg with a radial Gaussianization loss to better achieve maximum entropy.

Findings

Improves representation diversity and informativeness.

Consistently outperforms VCReg on synthetic and real datasets.

Effectively reduces higher-order dependencies in features.

Abstract

Self-supervised learning aims to learn maximally informative representations, but explicit information maximization is hindered by the curse of dimensionality. Existing methods like VCReg address this by regularizing first and second-order feature statistics, which cannot fully achieve maximum entropy. We propose Radial-VCReg, which augments VCReg with a radial Gaussianization loss that aligns feature norms with the Chi distribution-a defining property of high-dimensional Gaussians. We prove that Radial-VCReg transforms a broader class of distributions towards normality compared to VCReg and show on synthetic and real-world datasets that it consistently improves performance by reducing higher-order dependencies and promoting more diverse and informative representations.