Geometry and Generalization: Eigenvalues as predictors of where a network will fail to generalize

TL;DR

This paper investigates how the eigenvalues of Jacobian matrices of trained autoencoders can predict their generalization failure, providing dataset-independent metrics based solely on model parameters.

Contribution

It introduces bounds on mean squared errors using eigenvector orthogonality assumptions and shows that eigenvalues' trace and product predict test error without dataset knowledge.

Findings

Eigenvalues of Jacobians correlate with generalization performance.

Trace and product of eigenvalues predict test MSE.

Model parameters alone suffice to assess generalization ability.

Abstract

We study the deformation of the input space by a trained autoencoder via the Jacobians of the trained weight matrices. In doing so, we prove bounds for the mean squared errors for points in the input space, under assumptions regarding the orthogonality of the eigenvectors. We also show that the trace and the product of the eigenvalues of the Jacobian matrices is a good predictor of the MSE on test points. This is a dataset independent means of testing an autoencoder's ability to generalize on new input. Namely, no knowledge of the dataset on which the network was trained is needed, only the parameters of the trained model.