Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain

TL;DR

This paper introduces Mecellem models, specialized Turkish legal language models developed via scratch pre-training and continual domain adaptation, achieving high retrieval performance and domain-specific understanding with efficient training strategies.

Contribution

It presents a novel framework for Turkish legal domain models, including a scratch-trained encoder with checkpoint selection and a continual pre-training decoder, both optimized for efficiency and performance.

Findings

Encoder models achieve top-3 Turkish retrieval leaderboard rankings.

The approach attains 92.36% production efficiency compared to state-of-the-art models.

Continual pre-training reduces perplexity by 36.2% on Turkish legal texts.

Abstract

This paper presents Mecellem models, a framework for developing specialized language models for the Turkish legal domain through domain adaptation strategies. We make two contributions: (1)Encoder Model Pre-trained from Scratch: ModernBERT-based bidirectional encoders pre-trained on a Turkish-dominant corpus of 112.7 billion tokens. We implement a checkpoint selection strategy that evaluates downstream retrieval performance throughout training, revealing that optimal checkpoints achieve best retrieval scores before pre-training loss reaches its minimum. Our encoder models achieve top-3 rankings on the Turkish retrieval leaderboard, with smaller models (155M parameters) achieving comparable performance to larger reference models (307M-567M parameters). Our approach achieves 92.36% production efficiency compared to state-of-the-art models (embeddinggemma-300m: 100.00%, BAAI/bge-m3: 99.54%, newmindai/bge-m3-stsb: 94.38%), ranking fourth overall despite requiring less computational resources. SOTA models rely on multi-stage, computationally intensive training pipelines, making our single-stage pre-training followed by efficient post-training approach a cost-effective alternative; (2)Decoder Model with Continual Pre-training (CPT): Qwen3-1.7B and Qwen3-4B models adapted to Turkish legal domain through controlled curriculum learning. Four-phase CPT with optimal sample ratios enables gradual transition from general language knowledge to specialized legal terminology and long-context reasoning. This approach achieves 36.2% perplexity reduction on Turkish legal text, demonstrating domain adaptation gains.