Pay Attention to Small Weights

TL;DR

This paper introduces NANOADAM, a finetuning method that selectively updates small-magnitude weights based on observed gradient-weight relationships, improving efficiency and performance in NLP and vision tasks.

Contribution

The paper proposes NANOADAM, a gradient-free, weight-based finetuning approach that preserves important large weights and enhances generalization.

Findings

NANOADAM outperforms standard methods in NLP and vision tasks.

Selective weight updating reduces resource usage.

Preserving large weights mitigates catastrophic forgetting.

Abstract

Finetuning large pretrained neural networks is known to be resource-intensive, both in terms of memory and computational cost. To mitigate this, a common approach is to restrict training to a subset of the model parameters. By analyzing the relationship between gradients and weights during finetuning, we observe a notable pattern: large gradients are often associated with small-magnitude weights. This correlation is more pronounced in finetuning settings than in training from scratch. Motivated by this observation, we propose NANOADAM, which dynamically updates only the small-magnitude weights during finetuning and offers several practical advantages: first, this criterion is gradient-free -- the parameter subset can be determined without gradient computation; second, it preserves large-magnitude weights, which are likely to encode critical features learned during pretraining, thereby reducing the risk of catastrophic forgetting; thirdly, it permits the use of larger learning rates and consistently leads to better generalization performance in experiments. We demonstrate this for both NLP and vision tasks.