Distributional Autoencoders Know the Score

TL;DR

The paper introduces the Distributional Principal Autoencoder (DPA), which provides theoretical guarantees for disentangling data factors and estimating intrinsic dimension, unifying distribution modeling and dimensionality estimation.

Contribution

It offers exact theoretical relations linking autoencoder geometry to data distribution scores and guarantees on disentanglement and intrinsic dimension estimation.

Findings

DPA's geometry relates to data distribution score, enabling disentanglement.

Exact recovery of score from samples is possible.

The model approximates the minimum free-energy path for specific distributions.

Abstract

The Distributional Principal Autoencoder (DPA) combines distributionally correct reconstruction with principal-component-like interpretability of the encodings. In this work, we provide exact theoretical guarantees on both fronts. First, we derive a closed-form relation linking each optimal level-set geometry to the data-distribution score. This result explains DPA's empirical ability to disentangle factors of variation of the data, as well as allows the score to be recovered directly from samples. When the data follows the Boltzmann distribution, we demonstrate that this relation yields an approximation of the minimum free-energy path for the Mueller-Brown potential in a single fit. Second, we prove that if the data lies on a manifold that can be approximated by the encoder, latent components beyond the manifold dimension are conditionally independent of the data distribution - carrying no additional information - and thus reveal the intrinsic dimension. Together, these results show that a single model can learn the data distribution and its intrinsic dimension with exact guarantees simultaneously, unifying two longstanding goals of unsupervised learning.