Apertus: Democratizing Open and Compliant LLMs for Global Language Environments

TL;DR

Apertus introduces fully open, multilingual large language models that prioritize data compliance and transparency, achieving competitive performance while providing comprehensive open artifacts for community use.

Contribution

Apertus is the first open LLM suite that ensures data compliance, filters harmful content, and expands multilingual coverage, with full transparency of development artifacts.

Findings

Approaches state-of-the-art results on multilingual benchmarks

Outperforms other open models in multilingual tasks

Provides transparent access to all development artifacts

Abstract

We present Apertus, a fully open suite of large language models (LLMs) designed to address two systemic shortcomings in today's open model ecosystem: data compliance and multilingual representation. Unlike many prior models that release weights without reproducible data pipelines or regard for content-owner rights, Apertus models are pretrained exclusively on openly available data, retroactively respecting `robots.txt` exclusions and filtering for non-permissive, toxic, and personally identifiable content. To mitigate risks of memorization, we adopt the Goldfish objective during pretraining, strongly suppressing verbatim recall of data while retaining downstream task performance. The Apertus models also expand multilingual coverage, training on 15T tokens from over 1800 languages, with ~40% of pretraining data allocated to non-English content. Released at 8B and 70B scales, Apertus approaches state-of-the-art results among fully open models on multilingual benchmarks, rivalling or surpassing open-weight counterparts. Beyond model weights, we release all scientific artifacts from our development cycle with a permissive license, including data preparation scripts, checkpoints, evaluation suites, and training code, enabling transparent audit and extension.