Language Resources for Dutch Large Language Modelling

TL;DR

This paper introduces Dutch-specific language models based on Llama 2, along with datasets and benchmarks, to address the gap in Dutch NLP resources and facilitate future research and development.

Contribution

It presents fine-tuned Dutch Llama 2 models, new datasets, a performance leaderboard, and a critical discussion on advancing Dutch language modeling.

Findings

Fine-tuned Dutch Llama 2 models outperform baseline models

New datasets improve Dutch language understanding and generation

Benchmark results highlight current strengths and gaps in Dutch NLP

Abstract

Despite the rapid expansion of types of large language models, there remains a notable gap in models specifically designed for the Dutch language. This gap is not only a shortage in terms of pretrained Dutch models but also in terms of data, and benchmarks and leaderboards. This work provides a small step to improve the situation. First, we introduce two fine-tuned variants of the Llama 2 13B model. We first fine-tuned Llama 2 using Dutch-specific web-crawled data and subsequently refined this model further on multiple synthetic instruction and chat datasets. These datasets as well as the model weights are made available. In addition, we provide a leaderboard to keep track of the performance of (Dutch) models on a number of generation tasks, and we include results of a number of state-of-the-art models, including our own. Finally we provide a critical conclusion on what we believe is needed to push forward Dutch language models and the whole eco-system around the models.