Meta-Learning Fast Weight Language Models

TL;DR

This paper introduces Fast Weight Layers (FWLs), a neural component that efficiently mimics dynamic evaluation in language models by using linear attention, enabling better performance with less computational cost.

Contribution

The paper proposes FWLs, a novel neural component that allows models to benefit from dynamic evaluation techniques more efficiently and can be integrated into existing transformers.

Findings

FWLs significantly improve language modeling perplexity.

FWLs require less compute and memory than traditional dynamic evaluation.

FWLs can be applied during training for better gradient utilization.

Abstract

Dynamic evaluation of language models (LMs) adapts model parameters at test time using gradient information from previous tokens and substantially improves LM performance. However, it requires over 3x more compute than standard inference. We present Fast Weight Layers (FWLs), a neural component that provides the benefits of dynamic evaluation much more efficiently by expressing gradient updates as linear attention. A key improvement over dynamic evaluation is that FWLs can also be applied at training time so the model learns to make good use of gradient updates. FWLs can easily be added on top of existing transformer models, require relatively little extra compute or memory to run, and significantly improve language modeling perplexity.