Evaluating the Translation Performance of Large Language Models Based on Euas-20

TL;DR

This paper introduces Euas-20, a dataset designed to evaluate large language models' translation performance across languages, highlighting the impact of pre-training data on their translation capabilities.

Contribution

The paper presents Euas-20, a new dataset for assessing LLMs' translation performance and analyzes how pre-training data influences translation abilities.

Findings

Euas-20 effectively evaluates LLM translation performance.

Pre-training data significantly affects translation quality.

Large language models show varied translation abilities across languages.

Abstract

In recent years, with the rapid development of deep learning technology, large language models (LLMs) such as BERT and GPT have achieved breakthrough results in natural language processing tasks. Machine translation (MT), as one of the core tasks of natural language processing, has also benefited from the development of large language models and achieved a qualitative leap. Despite the significant progress in translation performance achieved by large language models, machine translation still faces many challenges. Therefore, in this paper, we construct the dataset Euas-20 to evaluate the performance of large language models on translation tasks, the translation ability on different languages, and the effect of pre-training data on the translation ability of LLMs for researchers and developers.