HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model

TL;DR

Hebatron is a novel Hebrew-specialized large language model based on a sparse Mixture-of-Experts architecture, featuring a curriculum training approach, high inference throughput, and native long-context support.

Contribution

First open-weight Hebrew-specific MoE model using Nemotron-3 architecture with long-context support and curriculum training methodology.

Findings

Achieves 73.8% Hebrew reasoning accuracy

Outperforms existing Hebrew models like DictaLM-3.0-24B-Thinking

Provides 9x higher inference throughput at long contexts

Abstract

We present Hebatron, a Hebrew-specialized open-weight large language model built on the NVIDIA Nemotron-3 sparse Mixture-of-Experts architecture. Training employs a three-phase easy-to-hard curriculum with continuous anti-forgetting anchoring, followed by supervised fine-tuning on 2 million bilingual Hebrew--English samples. The curriculum ordering alone yields a 3-point aggregate benchmark gain over the reversed configuration. Hebatron achieves a Hebrew reasoning average of 73.8\%, outperforming DictaLM-3.0-24B-Thinking (68.9\%) and remaining competitive with Gemma-3-27B-IT on GSM8K-HE and Israeli Trivia, while activating only 3B parameters per forward pass across a 30B-parameter model, delivering approximately 9 times higher inference throughput at native context lengths up to 65,536 tokens. To our knowledge, this is the first language-specific adaptation of the Nemotron-3 architecture for any target language, and the first open-weight Hebrew-specialized MoE model with native long-context support. Model weights are released openly to support further research in Hebrew and Semitic-language NLP.