Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA

TL;DR

This paper introduces time-contrastive learning (TCL), a novel unsupervised deep learning approach leveraging nonstationary data structures, which achieves the first rigorous identifiability in nonlinear ICA models.

Contribution

It establishes a new principle for unsupervised feature learning from time series data using TCL, linking it to nonlinear ICA and proving its identifiability.

Findings

TCL enables optimal discrimination of time segments in data.

TCL combined with linear ICA estimates nonlinear ICA up to point-wise transformations.

The approach provides the first rigorous and general identifiability result for nonlinear ICA.

Abstract

Nonlinear independent component analysis (ICA) provides an appealing framework for unsupervised feature learning, but the models proposed so far are not identifiable. Here, we first propose a new intuitive principle of unsupervised deep learning from time series which uses the nonstationary structure of the data. Our learning principle, time-contrastive learning (TCL), finds a representation which allows optimal discrimination of time segments (windows). Surprisingly, we show how TCL can be related to a nonlinear ICA model, when ICA is redefined to include temporal nonstationarities. In particular, we show that TCL combined with linear ICA estimates the nonlinear ICA model up to point-wise transformations of the sources, and this solution is unique --- thus providing the first identifiability result for nonlinear ICA which is rigorous, constructive, as well as very general.