Memory-based Parameter Adaptation

TL;DR

Memory-based Parameter Adaptation enhances neural networks by storing examples in memory and using context-based lookup to directly modify weights, enabling rapid adaptation and overcoming limitations like catastrophic forgetting.

Contribution

The paper introduces a memory-based method for neural network adaptation that allows fast, local weight updates using stored examples, addressing key limitations of traditional training.

Findings

Enables high learning rates for quick adaptation.

Reduces catastrophic forgetting in neural networks.

Improves learning with imbalanced classes and during evaluation.

Abstract

Deep neural networks have excelled on a wide range of problems, from vision to language and game playing. Neural networks very gradually incorporate information into weights as they process data, requiring very low learning rates. If the training distribution shifts, the network is slow to adapt, and when it does adapt, it typically performs badly on the training distribution before the shift. Our method, Memory-based Parameter Adaptation, stores examples in memory and then uses a context-based lookup to directly modify the weights of a neural network. Much higher learning rates can be used for this local adaptation, reneging the need for many iterations over similar data before good predictions can be made. As our method is memory-based, it alleviates several shortcomings of neural networks, such as catastrophic forgetting, fast, stable acquisition of new knowledge, learning with an imbalanced class labels, and fast learning during evaluation. We demonstrate this on a range of supervised tasks: large-scale image classification and language modelling.