Adversarial Dependence Minimization

TL;DR

This paper introduces a scalable, differentiable adversarial algorithm that minimizes nonlinear dependencies among features, enhancing representation learning and generalization in machine learning models.

Contribution

It presents a novel adversarial dependence minimization method that extends beyond linear decorrelation, applicable to various machine learning tasks.

Findings

Effective in reducing nonlinear dependencies among features

Improves generalization in image classification

Prevents dimensional collapse in self-supervised learning

Abstract

Many machine learning techniques rely on minimizing the covariance between output feature dimensions to extract minimally redundant representations from data. However, these methods do not eliminate all dependencies/redundancies, as linearly uncorrelated variables can still exhibit nonlinear relationships. This work provides a differentiable and scalable algorithm for dependence minimization that goes beyond linear pairwise decorrelation. Our method employs an adversarial game where small networks identify dependencies among feature dimensions, while the encoder exploits this information to reduce dependencies. We provide empirical evidence of the algorithm's convergence and demonstrate its utility in three applications: extending PCA to nonlinear decorrelation, improving the generalization of image classification methods, and preventing dimensional collapse in self-supervised representation learning.