EuroLLM-22B: Technical Report

TL;DR

EuroLLM-22B is a large multilingual language model designed specifically for European languages, addressing underrepresentation issues and demonstrating strong performance across various benchmarks.

Contribution

This paper introduces EuroLLM-22B, a multilingual model trained from scratch to support all EU languages and additional languages, with comprehensive development details and released resources.

Findings

EuroLLM-22B performs well on reasoning, instruction following, and translation benchmarks.

The model achieves results competitive with similar-sized models.

Resources and datasets are publicly released for future research.

Abstract

This report presents EuroLLM-22B, a large language model trained from scratch to support the needs of European citizens by covering all 24 official European Union languages and 11 additional languages. EuroLLM addresses the issue of European languages being underrepresented and underserved in existing open large language models. We provide a comprehensive overview of EuroLLM-22B's development, including tokenizer design, architectural specifications, data filtering, and training procedures. Across a broad set of multilingual benchmarks, EuroLLM-22B demonstrates strong performance in reasoning, instruction following, and translation, achieving results competitive with models of comparable size. To support future research, we release our base and instruction-tuned models, our multilingual web pretraining data and updated EuroBlocks instruction datasets, as well as our pre-training and evaluation codebases.