Extreme Memorization via Scale of Initialization
Harsh Mehta, Ashok Cutkosky, Behnam Neyshabur

TL;DR
This paper investigates how the scale of initialization in neural networks influences their tendency to memorize training data versus generalize, revealing that larger scales can lead to extreme memorization depending on activation and loss functions.
Contribution
It introduces an experimental framework showing the impact of initialization scale on implicit regularization and proposes an alignment measure correlating with generalization performance.
Findings
Increasing initialization scale leads to more memorization.
Activation and loss functions affect generalization behavior.
An alignment measure correlates with model generalization.
Abstract
We construct an experimental setup in which changing the scale of initialization strongly impacts the implicit regularization induced by SGD, interpolating from good generalization performance to completely memorizing the training set while making little progress on the test set. Moreover, we find that the extent and manner in which generalization ability is affected depends on the activation and loss function used, with activation demonstrating extreme memorization. In the case of the homogeneous ReLU activation, we show that this behavior can be attributed to the loss function. Our empirical investigation reveals that increasing the scale of initialization correlates with misalignment of representations and gradients across examples in the same class. This insight allows us to devise an alignment measure over gradients and representations which can capture this phenomenon. We…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Stochastic Gradient Optimization Techniques · Neural Networks and Applications
Methods*Communicated@Fast*How Do I Communicate to Expedia? · Stochastic Gradient Descent
