MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation

TL;DR

MrBERT is a versatile multilingual encoder family that achieves state-of-the-art results in specific languages and domains while optimizing for efficiency through flexible vector representations.

Contribution

Introduces MrBERT, a multilingual encoder family with adaptive vocabulary, domain-specific tuning, and efficient vector representations for improved performance and deployment.

Findings

State-of-the-art results on Catalan and Spanish tasks

Robust performance in biomedical and legal domains

Reduced inference and storage costs

Abstract

We introduce MrBERT, a family of 150M-300M parameter encoders built on the ModernBERT architecture and pre-trained on 35 languages and code. Through targeted adaptation, this model family achieves state-of-the-art results on Catalan- and Spanish-specific tasks, while establishing robust performance across specialized biomedical and legal domains. To bridge the gap between research and production, we incorporate Matryoshka Representation Learning (MRL), enabling flexible vector sizing that significantly reduces inference and storage costs. Ultimately, the MrBERT family demonstrates that modern encoder architectures can be optimized for both localized linguistic excellence and efficient, high-stakes domain specialization. We open source the complete model family on Huggingface.