Dependence Induced Representations

TL;DR

This paper explores how to learn feature representations based on the dependence between variables, connecting them to maximal correlation and minimal sufficient statistics, and introduces a new learning design with hyperparameter tuning.

Contribution

It provides necessary and sufficient conditions for dependence induced representations, links them to maximal correlation functions, and offers a new learning framework with hyperparameter tuning during inference.

Findings

Features can be expressed as a composition of a loss-dependent function and the maximal correlation function.

Provides a statistical interpretation of neural collapse in deep classifiers.

Introduces a learning design based on feature separation with hyperparameter tuning.

Abstract

We study the problem of learning feature representations from a pair of random variables, where we focus on the representations that are induced by their dependence. We provide sufficient and necessary conditions for such dependence induced representations, and illustrate their connections to Hirschfeld--Gebelein--R\'{e}nyi (HGR) maximal correlation functions and minimal sufficient statistics. We characterize a large family of loss functions that can learn dependence induced representations, including cross entropy, hinge loss, and their regularized variants. In particular, we show that the features learned from this family can be expressed as the composition of a loss-dependent function and the maximal correlation function, which reveals a key connection between representations learned from different losses. Our development also gives a statistical interpretation of the neural collapse phenomenon observed in deep classifiers. Finally, we present the learning design based on the feature separation, which allows hyperparameter tuning during inference.