BERTje: A Dutch BERT Model

TL;DR

BERTje is a Dutch-specific BERT model trained on a large diverse dataset, outperforming multilingual BERT on multiple NLP tasks and publicly available for research use.

Contribution

This paper introduces BERTje, a monolingual Dutch BERT model trained on 2.4 billion tokens, demonstrating improved performance over multilingual BERT on various NLP tasks.

Findings

BERTje outperforms multilingual BERT on Dutch NLP tasks

BERTje is trained on a larger, more diverse dataset

The model is publicly available for research use

Abstract

The transformer-based pre-trained language model BERT has helped to improve state-of-the-art performance on many natural language processing (NLP) tasks. Using the same architecture and parameters, we developed and evaluated a monolingual Dutch BERT model called BERTje. Compared to the multilingual BERT model, which includes Dutch but is only based on Wikipedia text, BERTje is based on a large and diverse dataset of 2.4 billion tokens. BERTje consistently outperforms the equally-sized multilingual BERT model on downstream NLP tasks (part-of-speech tagging, named-entity recognition, semantic role labeling, and sentiment analysis). Our pre-trained Dutch BERT model is made available at https://github.com/wietsedv/bertje.