HOME: High-Order Mixed-Moment-based Embedding for Representation Learning

TL;DR

This paper introduces HOME, a novel high-order mixed-moment-based embedding approach that reduces redundancy among multiple feature variables in self-supervised learning, outperforming existing pairwise methods.

Contribution

HOME is the first method to utilize high-order statistics for reducing redundancy in representation learning, capturing complex inter-variable dependencies.

Findings

HOME outperforms Barlow Twins in linear evaluation tasks.

A three-order HOME scheme significantly improves feature independence.

The approach leverages multivariate mutual information for better feature disentanglement.

Abstract

Minimum redundancy among different elements of an embedding in a latent space is a fundamental requirement or major preference in representation learning to capture intrinsic informational structures. Current self-supervised learning methods minimize a pair-wise covariance matrix to reduce the feature redundancy and produce promising results. However, such representation features of multiple variables may contain the redundancy among more than two feature variables that cannot be minimized via the pairwise regularization. Here we propose the High-Order Mixed-Moment-based Embedding (HOME) strategy to reduce the redundancy between any sets of feature variables, which is to our best knowledge the first attempt to utilize high-order statistics/information in this context. Multivariate mutual information is minimum if and only if multiple variables are mutually independent, which suggests the necessary conditions of factorized mixed moments among multiple variables. Based on these statistical and information theoretic principles, our general HOME framework is presented for self-supervised representation learning. Our initial experiments show that a simple version in the form of a three-order HOME scheme already significantly outperforms the current two-order baseline method (i.e., Barlow Twins) in terms of the linear evaluation on representation features.