AraModernBERT: Transtokenized Initialization and Long-Context Encoder Modeling for Arabic

TL;DR

AraModernBERT introduces transtokenized initialization and long-context encoding for Arabic, significantly improving language modeling and downstream NLP task performance for Arabic language processing.

Contribution

This work adapts ModernBERT architecture to Arabic, emphasizing transtokenization and long-context modeling, which were previously underexplored for non-English languages.

Findings

Transtokenization is crucial for Arabic language modeling.

Enhanced long-context modeling improves intrinsic language understanding.

Strong transfer to various Arabic NLP tasks demonstrated.

Abstract

Encoder-only transformer models remain widely used for discriminative NLP tasks, yet recent architectural advances have largely focused on English. In this work, we present AraModernBERT, an adaptation of the ModernBERT encoder architecture to Arabic, and study the impact of transtokenized embedding initialization and native long-context modeling up to 8,192 tokens. We show that transtokenization is essential for Arabic language modeling, yielding dramatic improvements in masked language modeling performance compared to non-transtokenized initialization. We further demonstrate that AraModernBERT supports stable and effective long-context modeling, achieving improved intrinsic language modeling performance at extended sequence lengths. Downstream evaluations on Arabic natural language understanding tasks, including inference, offensive language detection, question-question similarity, and named entity recognition, confirm strong transfer to discriminative and sequence labeling settings. Our results highlight practical considerations for adapting modern encoder architectures to Arabic and other languages written in Arabic-derived scripts.