MobileLLM-Pro Technical Report

TL;DR

MobileLLM-Pro is a 1-billion-parameter on-device language model that achieves state-of-the-art performance on multiple benchmarks, supports very long contexts, and maintains efficiency through innovative techniques like implicit positional distillation and model merging.

Contribution

The paper introduces MobileLLM-Pro, a novel 1-billion-parameter language model optimized for on-device deployment with four key innovations for long context support and efficiency.

Findings

Outperforms Gemma 3-1B and Llama 3.2-1B on 11 benchmarks

Supports context windows up to 128,000 tokens

Maintains performance with 4-bit quantization

Abstract

Efficient on-device language models around 1 billion parameters are essential for powering low-latency AI applications on mobile and wearable devices. However, achieving strong performance in this model class, while supporting long context windows and practical deployment remains a significant challenge. We introduce MobileLLM-Pro, a 1-billion-parameter language model optimized for on-device deployment. MobileLLM-Pro achieves state-of-the-art results across 11 standard benchmarks, significantly outperforming both Gemma 3-1B and Llama 3.2-1B, while supporting context windows of up to 128,000 tokens and showing only minor performance regressions at 4-bit quantization. These improvements are enabled by four core innovations: (1) implicit positional distillation, a novel technique that effectively instills long-context capabilities through knowledge distillation; (2) a specialist model merging framework that fuses multiple domain experts into a compact model without parameter growth; (3) simulation-driven data mixing using utility estimation; and (4) 4-bit quantization-aware training with self-distillation. We release our model weights and code to support future research in efficient on-device language models.