Contamination Report for Multilingual Benchmarks

TL;DR

This paper investigates the contamination of popular multilingual benchmarks in large language models, revealing widespread issues that affect the reliability of multilingual evaluation results.

Contribution

It introduces a systematic analysis of benchmark contamination in multilingual LLMs using the Black Box test, highlighting prevalent data leakage problems.

Findings

Almost all tested models show signs of benchmark contamination.

Most benchmarks are contaminated across multiple models.

Contamination impacts the validity of multilingual evaluation results.

Abstract

Benchmark contamination refers to the presence of test datasets in Large Language Model (LLM) pre-training or post-training data. Contamination can lead to inflated scores on benchmarks, compromising evaluation results and making it difficult to determine the capabilities of models. In this work, we study the contamination of popular multilingual benchmarks in LLMs that support multiple languages. We use the Black Box test to determine whether $7$ frequently used multilingual benchmarks are contaminated in $7$ popular open and closed LLMs and find that almost all models show signs of being contaminated with almost all the benchmarks we test. Our findings can help the community determine the best set of benchmarks to use for multilingual evaluation.