Neural PCA for Flow-Based Representation Learning

TL;DR

This paper introduces Neural-PCA, a method for extracting principal components from invertible generative models like normalizing flows, improving downstream task performance without labels by capturing the most informative features.

Contribution

Neural-PCA is a novel approach that operates in full dimensionality to recover principal components in descending order, enhancing representation quality for downstream tasks.

Findings

Achieves 5-10% performance improvement in downstream tasks.

Effectively captures the most informative features without label supervision.

Performance gains are consistent regardless of trailing dimension count.

Abstract

Of particular interest is to discover useful representations solely from observations in an unsupervised generative manner. However, the question of whether existing normalizing flows provide effective representations for downstream tasks remains mostly unanswered despite their strong ability for sample generation and density estimation. This paper investigates this problem for such a family of generative models that admits exact invertibility. We propose Neural Principal Component Analysis (Neural-PCA) that operates in full dimensionality while capturing principal components in \emph{descending} order. Without exploiting any label information, the principal components recovered store the most informative elements in their \emph{leading} dimensions and leave the negligible in the \emph{trailing} ones, allowing for clear performance improvements of $5\%$-$10\%$ in downstream tasks. Such improvements are empirically found consistent irrespective of the number of latent trailing dimensions dropped. Our work suggests that necessary inductive bias be introduced into generative modelling when representation quality is of interest.