TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish

TL;DR

TabiBERT is a large-scale, modern Turkish language model that outperforms previous models across multiple tasks, supported by a comprehensive benchmark and trained on a vast, diverse corpus.

Contribution

This work introduces TabiBERT, the first monolingual ModernBERT-based Turkish encoder trained from scratch on a large, multi-domain corpus, and presents TabiBench, a standardized benchmark for Turkish NLP.

Findings

TabiBERT achieves 77.58 on TabiBench, surpassing BERTurk by 1.62 points.

It attains up to 2.65x inference speedup and reduced GPU memory usage.

Outperforms task-specific models like TurkishBERTweet on average by 1.47 points.

Abstract

Since the inception of BERT, encoder-only Transformers have evolved significantly in computational efficiency, training stability, and long-context modeling. ModernBERT consolidates these advances by integrating Rotary Positional Embeddings (RoPE), FlashAttention, and refined normalization. Despite these developments, Turkish NLP lacks a monolingual encoder trained from scratch, incorporating such modern architectural paradigms. This work introduces TabiBERT, a monolingual Turkish encoder based on ModernBERT architecture trained from scratch on a large, curated corpus. TabiBERT is pre-trained on one trillion tokens sampled from an 84.88B token multi-domain corpus: web text (73%), scientific publications (20%), source code (6%), and mathematical content (0.3%). It supports 8,192-token context length (16x original BERT), achieves up to 2.65x inference speedup, and reduces GPU memory consumption, enabling larger batch sizes. We introduce TabiBench with 28 datasets across eight task categories with standardized splits and protocols, evaluated using GLUE-style macro-averaging. TabiBERT attains 77.58 on TabiBench, outperforming BERTurk by 1.62 points and establishing state-of-the-art on five of eight categories, with particularly strong gains on question answering (+9.55 points), code retrieval (+2.41 points), and academic understanding (+0.66 points). Compared with task-specific prior best results, including specialized models like TurkishBERTweet, TabiBERT achieves +1.47 average improvement, indicating robust cross-domain generalization. We release model weights, training configurations, and evaluation code for transparent, reproducible Turkish encoder research.