Geometric instability of out of distribution data across autoencoder architecture

TL;DR

This paper investigates the geometric stability of autoencoders trained on MNIST when evaluated on out-of-distribution data, revealing significant instability in their latent representations despite similar reconstructions.

Contribution

It introduces eigenvalue analysis of Jacobians to quantify the geometric instability of autoencoders across different data distributions and architectures.

Findings

Autoencoders reconstruct out-of-distribution data as similar generalized characters.

Eigenvalues indicate instability in latent representations for OOD data.

Instability persists across different autoencoder architectures.

Abstract

We study the map learned by a family of autoencoders trained on MNIST, and evaluated on ten different data sets created by the random selection of pixel values according to ten different distributions. Specifically, we study the eigenvalues of the Jacobians defined by the weight matrices of the autoencoder at each training and evaluation point. For high enough latent dimension, we find that each autoencoder reconstructs all the evaluation data sets as similar \emph{generalized characters}, but that this reconstructed \emph{generalized character} changes across autoencoder. Eigenvalue analysis shows that even when the reconstructed image appears to be an MNIST character for all out of distribution data sets, not all have latent representations that are close to the latent representation of MNIST characters. All told, the eigenvalue analysis demonstrated a great deal of geometric instability of the autoencoder both as a function on out of distribution inputs, and across architectures on the same set of inputs.