GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Data

TL;DR

This paper introduces GemMaroc, a minimal-data, efficient approach to adapt large language models for Moroccan Arabic (Darija), achieving high performance without extensive compute or English regression.

Contribution

It presents a quality-over-quantity alignment strategy and a low-resource fine-tuning method to enhance Darija proficiency in LLMs while preserving reasoning skills.

Findings

GemMaroc-3-4B improves Darija MMLU scores from 32.8 to 42.7

Scaling to GemMaroc-27B matches or exceeds existing models on Darija benchmarks

Model training requires only 48 GPU hours, demonstrating efficiency and sustainability

Abstract

Open-source large language models (LLMs) still marginalise Moroccan Arabic (Darija), forcing practitioners either to bolt on heavyweight Arabic adapters or to sacrifice the very reasoning skills that make LLMs useful. We show that a rigorously quality-over-quantity alignment strategy can surface fluent Darija while safeguarding the backbone s cross-lingual reasoning at a sliver of the usual compute. We translate three compact instruction suites LIMA 1 K, DEITA 6 K and TULU 50 K into Darija, preserve 20 of the English originals, and add mathematics, coding and scientific prompts. A LoRA-tuned Gemma 3-4B trained on 5 K mixed instructions lifts DarijaMMLU from 32.8 to 42.7 ; adding the reasoning-dense TULU portion pushes it to 47.5 with no English regression. Scaling the identical recipe to Gemma 3-27B produces GemMaroc-27B, which matches Atlas-Chat on DarijaMMLU (61.6 ) and leaps ahead on Darija commonsense, scoring 60.5 on HellaSwag versus Atlas-Chat s 48.4 . Crucially, GemMaroc retains Gemma-27B s strong maths and general-reasoning ability, showing only minimal movement on GSM8K and English benchmarks. The entire model is trained in just 48 GPU.h, underscoring a Green AI pathway to inclusive, sustainable language technology. We release code, data and checkpoints to spur Darija-centric applications in education, public services and everyday digital interaction.