Pretraining Finnish ModernBERTs

TL;DR

This paper introduces Finnish ModernBERT models of various sizes, demonstrating their competitive performance and advantages over existing models, especially on long-context tasks, with empirical evaluations and publicly available code.

Contribution

Pretraining and releasing Finnish ModernBERT models in multiple sizes, highlighting their superior performance on long-context tasks and analyzing data usage effects.

Findings

Models outperform existing multilingual models

Superior on tasks requiring long contexts

Empirical analysis of data in final training stage

Abstract

This paper reports on pretraining ModernBERT encoder models in six different sizes, ranging from 51M to 475M parameters, with a focus on limited multilingualism, emphasizing languages relevant to Finland. Our models are competitive with, or superior to, existing multilingual models. They outperform monolingual models on tasks that require a context longer than 512 tokens. We present empirical results on using different data in the final stage of training. The code and models are publicly released.