Improving Language Plasticity via Pretraining with Active Forgetting

TL;DR

This paper introduces an active forgetting mechanism during pretraining of language models, enabling faster adaptation to new languages and better performance in low-data scenarios, especially for distant languages.

Contribution

It proposes resetting the embedding layer periodically during pretraining to enhance language plasticity, a novel approach for efficient multilingual adaptation.

Findings

Faster convergence in language adaptation tasks.

Outperforms standard models in low-data regimes.

Improves adaptation for distant languages.

Abstract

Pretrained language models (PLMs) are today the primary model for natural language processing. Despite their impressive downstream performance, it can be difficult to apply PLMs to new languages, a barrier to making their capabilities universally accessible. While prior work has shown it possible to address this issue by learning a new embedding layer for the new language, doing so is both data and compute inefficient. We propose to use an active forgetting mechanism during pretraining, as a simple way of creating PLMs that can quickly adapt to new languages. Concretely, by resetting the embedding layer every K updates during pretraining, we encourage the PLM to improve its ability of learning new embeddings within a limited number of updates, similar to a meta-learning effect. Experiments with RoBERTa show that models pretrained with our forgetting mechanism not only demonstrate faster convergence during language adaptation but also outperform standard ones in a low-data regime, particularly for languages that are distant from English.