When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluation

TL;DR

This paper investigates how large language models trained on specific benchmarks can memorize and transfer this knowledge across languages and directions, leading to inflated translation performance scores.

Contribution

It reveals cross-directional contamination in multilingual LLMs and proposes a method to probe memorization effects despite perturbations.

Findings

Bloomz shows contamination in FLORES benchmark.

Contamination can transfer across translation directions.

Named entity replacement reduces BLEU, indicating memorization probing.

Abstract

Large language models (LLMs) can be benchmark-contaminated, resulting in inflated scores that mask memorization as generalization, and in multilingual settings, this memorization can even transfer to "uncontaminated" languages. Using the FLORES-200 translation benchmark as a diagnostic, we study two 7-8B instruction-tuned multilingual LLMs: Bloomz, which was trained on FLORES, and Llama as an uncontaminated control. We confirm Bloomz's FLORES contamination and demonstrate that machine translation contamination can be cross-directional, artificially boosting performance in unseen translation directions due to target-side memorization. Further analysis shows that recall of memorized references often persists despite various source-side perturbation efforts like paraphrasing and named entity replacement. However, replacing named entities leads to a consistent decrease in BLEU, suggesting an effective probing method for memorization in contaminated models.