Decoding Generalization from Memorization in Deep Neural Networks

TL;DR

This paper investigates why overparameterized deep neural networks memorize labels but still generalize well, providing evidence that certain layers retain latent information for true labels, which can be decoded to understand generalization.

Contribution

The study demonstrates that models retain latent information for true labels in their representations and introduces a technique to decode this information from trained models.

Findings

Models retain information for true labels in their internal representations.

Decoding techniques can extract latent generalization information from trained networks.

Empirical evidence shows layers preserve potential for better generalization.

Abstract

Overparameterized deep networks that generalize well have been key to the dramatic success of deep learning in recent years. The reasons for their remarkable ability to generalize are not well understood yet. When class labels in the training set are shuffled to varying degrees, it is known that deep networks can still reach perfect training accuracy at the detriment of generalization to true labels -- a phenomenon that has been called memorization. It has, however, been unclear why the poor generalization to true labels that accompanies such memorization, comes about. One possibility is that during training, all layers of the network irretrievably re-organize their representations in a manner that makes generalization to true labels difficult. The other possibility is that one or more layers of the trained network retain significantly more latent ability to generalize to true labels, but the network somehow "chooses" to readout in a manner that is detrimental to generalization to true labels. Here, we provide evidence for the latter possibility by demonstrating, empirically, that such models possess information in their representations for substantially-improved generalization to true labels. Furthermore, such abilities can be easily decoded from the internals of the trained model, and we build a technique to do so. We demonstrate results on multiple models trained with standard datasets. Our code is available at: https://github.com/simranketha/MASC_DNN.