Learning Ordered Representations in Latent Space for Intrinsic Dimension Estimation via Principal Component Autoencoder

TL;DR

This paper introduces a novel autoencoder framework that combines non-uniform variance regularization with an isometric constraint, effectively generalizing PCA to nonlinear settings for better intrinsic dimension estimation.

Contribution

It proposes a new autoencoder design that preserves PCA-like ordered representations in nonlinear latent spaces, improving nonlinear dimensionality reduction.

Findings

Successfully captures ordered principal components in nonlinear autoencoders

Enhances intrinsic dimension estimation accuracy

Maintains variance retention similar to PCA

Abstract

Autoencoders have long been considered a nonlinear extension of Principal Component Analysis (PCA). Prior studies have demonstrated that linear autoencoders (LAEs) can recover the ordered, axis-aligned principal components of PCA by incorporating non-uniform $\ell_2$ regularization or by adjusting the loss function. However, these approaches become insufficient in the nonlinear setting, as the remaining variance cannot be properly captured independently of the nonlinear mapping. In this work, we propose a novel autoencoder framework that integrates non-uniform variance regularization with an isometric constraint. This design serves as a natural generalization of PCA, enabling the model to preserve key advantages, such as ordered representations and variance retention, while remaining effective for nonlinear dimensionality reduction tasks.