Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning

TL;DR

This paper introduces a flexible framework for nonlinear ICA that leverages auxiliary variables and contrastive learning, providing theoretical guarantees and practical algorithms for recovering latent variables.

Contribution

It generalizes existing nonlinear ICA models by incorporating auxiliary variables and contrastive learning, with proven identifiability and consistency.

Findings

The framework guarantees identifiability of latent variables.

Algorithmic implementation via logistic regression is effective.

The approach unifies and extends previous nonlinear ICA methods.

Abstract

Nonlinear ICA is a fundamental problem for unsupervised representation learning, emphasizing the capacity to recover the underlying latent variables generating the data (i.e., identifiability). Recently, the very first identifiability proofs for nonlinear ICA have been proposed, leveraging the temporal structure of the independent components. Here, we propose a general framework for nonlinear ICA, which, as a special case, can make use of temporal structure. It is based on augmenting the data by an auxiliary variable, such as the time index, the history of the time series, or any other available information. We propose to learn nonlinear ICA by discriminating between true augmented data, or data in which the auxiliary variable has been randomized. This enables the framework to be implemented algorithmically through logistic regression, possibly in a neural network. We provide a comprehensive proof of the identifiability of the model as well as the consistency of our estimation method. The approach not only provides a general theoretical framework combining and generalizing previously proposed nonlinear ICA models and algorithms, but also brings practical advantages.