Towards Multilingual LLM Evaluation for European Languages

TL;DR

This paper presents a multilingual evaluation framework for European languages, using translated benchmarks to assess 40 LLMs across 21 languages, addressing the challenge of multilingual performance measurement.

Contribution

It introduces a new multilingual evaluation approach with translated benchmarks and datasets, enabling consistent assessment of LLMs in European languages.

Findings

Effectiveness of translated benchmarks assessed

Impact of different translation services evaluated

Framework and datasets made publicly available

Abstract

The rise of Large Language Models (LLMs) has revolutionized natural language processing across numerous languages and tasks. However, evaluating LLM performance in a consistent and meaningful way across multiple European languages remains challenging, especially due to the scarcity of language-parallel multilingual benchmarks. We introduce a multilingual evaluation approach tailored for European languages. We employ translated versions of five widely-used benchmarks to assess the capabilities of 40 LLMs across 21 European languages. Our contributions include examining the effectiveness of translated benchmarks, assessing the impact of different translation services, and offering a multilingual evaluation framework for LLMs that includes newly created datasets: EU20-MMLU, EU20-HellaSwag, EU20-ARC, EU20-TruthfulQA, and EU20-GSM8K. The benchmarks and results are made publicly available to encourage further research in multilingual LLM evaluation.