RobBERT-2022: Updating a Dutch Language Model to Account for Evolving Language Use

TL;DR

This paper updates the Dutch RobBERT language model by incorporating recent vocabulary and further pre-training, demonstrating improved performance on language tasks that benefit from current language use.

Contribution

The paper introduces a method for updating a pre-trained language model with new vocabulary and additional training to better reflect evolving language.

Findings

Significant performance improvements on certain language tasks

Updated tokenizer with new high-frequency tokens

Demonstrates benefits of continual language model updating

Abstract

Large transformer-based language models, e.g. BERT and GPT-3, outperform previous architectures on most natural language processing tasks. Such language models are first pre-trained on gigantic corpora of text and later used as base-model for finetuning on a particular task. Since the pre-training step is usually not repeated, base models are not up-to-date with the latest information. In this paper, we update RobBERT, a RoBERTa-based state-of-the-art Dutch language model, which was trained in 2019. First, the tokenizer of RobBERT is updated to include new high-frequent tokens present in the latest Dutch OSCAR corpus, e.g. corona-related words. Then we further pre-train the RobBERT model using this dataset. To evaluate if our new model is a plug-in replacement for RobBERT, we introduce two additional criteria based on concept drift of existing tokens and alignment for novel tokens.We found that for certain language tasks this update results in a significant performance increase. These results highlight the benefit of continually updating a language model to account for evolving language use.