Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs

TL;DR

This paper introduces two multilingual large language models, Teuken 7B-base and Teuken 7B-instruct, designed to support all 24 EU languages, demonstrating strong multilingual performance and addressing limitations of English-centric models.

Contribution

The paper presents new multilingual LLMs supporting all EU languages, with tailored data, tokenizer, and training methods, filling a gap in European language AI resources.

Findings

Strong performance on European multilingual benchmarks

Supports all 24 EU official languages

Addresses English-centric bias in existing LLMs

Abstract

We present two multilingual LLMs, Teuken 7B-base and Teuken 7B-instruct, designed to embrace Europe's linguistic diversity by supporting all 24 official languages of the European Union. Trained on a dataset comprising around 60% non-English data and utilizing a custom multilingual tokenizer, our models address the limitations of existing LLMs that predominantly focus on English or a few high-resource languages. We detail the models' development principles, i.e., data composition, tokenizer optimization, and training methodologies. The models demonstrate strong performance across multilingual benchmarks, as evidenced by their performance on European versions of ARC, HellaSwag, and TruthfulQA.