Eigenvalues of Autoencoders in Training and at Initialization

TL;DR

This paper analyzes how the eigenvalue distribution of autoencoders' Jacobian matrices evolves during training on MNIST, revealing rapid convergence to trained eigenvalue profiles even after few epochs and comparing initial eigenvalues to theoretical models.

Contribution

It provides the first detailed study of eigenvalue evolution in autoencoders during training, linking empirical observations to random matrix theory.

Findings

Eigenvalue distributions differ significantly between untrained and trained autoencoders.

Eigenvalue distributions rapidly resemble those of fully trained autoencoders within early epochs.

Comparison of initial eigenvalues with theoretical models of random matrices.

Abstract

In this paper, we investigate the evolution of autoencoders near their initialization. In particular, we study the distribution of the eigenvalues of the Jacobian matrices of autoencoders early in the training process, training on the MNIST data set. We find that autoencoders that have not been trained have eigenvalue distributions that are qualitatively different from those which have been trained for a long time ($>$100 epochs). Additionally, we find that even at early epochs, these eigenvalue distributions rapidly become qualitatively similar to those of the fully trained autoencoders. We also compare the eigenvalues at initialization to pertinent theoretical work on the eigenvalues of random matrices and the products of such matrices.