Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization

TL;DR

This paper demonstrates that data whitening and certain second order optimization methods can impair or prevent generalization in machine learning models by reducing accessible information in the dataset, with some regularized approaches offering practical tradeoffs.

Contribution

It proves that whitening and specific second order methods limit information used for generalization, and shows how regularization can mitigate these effects.

Findings

Whitening and second order optimization reduce information for generalization.

Regularized second order optimization can accelerate training and sometimes improve generalization.

Experimental results confirm theoretical predictions across various architectures.

Abstract

Machine learning is predicated on the concept of generalization: a model achieving low error on a sufficiently large training set should also perform well on novel samples from the same distribution. We show that both data whitening and second order optimization can harm or entirely prevent generalization. In general, model training harnesses information contained in the sample-sample second moment matrix of a dataset. For a general class of models, namely models with a fully connected first layer, we prove that the information contained in this matrix is the only information which can be used to generalize. Models trained using whitened data, or with certain second order optimization schemes, have less access to this information, resulting in reduced or nonexistent generalization ability. We experimentally verify these predictions for several architectures, and further demonstrate that generalization continues to be harmed even when theoretical requirements are relaxed. However, we also show experimentally that regularized second order optimization can provide a practical tradeoff, where training is accelerated but less information is lost, and generalization can in some circumstances even improve.