The birth of Romanian BERT

TL;DR

Romanian BERT is the first dedicated Romanian transformer model, trained on a large corpus, with open-source tools and extensive evaluation, aiming to improve NLP tasks in Romanian.

Contribution

This paper introduces the first Romanian-specific BERT model, including corpus creation, training, evaluation, and open-source resources for practical use.

Findings

The Romanian BERT model performs well on various Romanian NLP datasets.

Open-source tools facilitate model fine-tuning and deployment.

The model advances Romanian NLP capabilities.

Abstract

Large-scale pretrained language models have become ubiquitous in Natural Language Processing. However, most of these models are available either in high-resource languages, in particular English, or as multilingual models that compromise performance on individual languages for coverage. This paper introduces Romanian BERT, the first purely Romanian transformer-based language model, pretrained on a large text corpus. We discuss corpus composition and cleaning, the model training process, as well as an extensive evaluation of the model on various Romanian datasets. We open source not only the model itself, but also a repository that contains information on how to obtain the corpus, fine-tune and use this model in production (with practical examples), and how to fully replicate the evaluation process.