Do GPTs Produce Less Literal Translations?

TL;DR

This paper compares the literalness of translations produced by GPTs and neural machine translation models, finding GPTs tend to produce less literal translations, especially with idiomatic expressions, while maintaining comparable or better quality.

Contribution

It introduces a systematic analysis of the literalness of GPT-generated translations versus NMT, highlighting differences in handling idiomatic language.

Findings

GPTs produce less literal translations than NMTs.

GPT translations have similar or better quality scores.

Differences are more pronounced with idiomatic expressions.

Abstract

Large Language Models (LLMs) such as GPT-3 have emerged as general-purpose language models capable of addressing many natural language generation or understanding tasks. On the task of Machine Translation (MT), multiple works have investigated few-shot prompting mechanisms to elicit better translations from LLMs. However, there has been relatively little investigation on how such translations differ qualitatively from the translations generated by standard Neural Machine Translation (NMT) models. In this work, we investigate these differences in terms of the literalness of translations produced by the two systems. Using literalness measures involving word alignment and monotonicity, we find that translations out of English (E-X) from GPTs tend to be less literal, while exhibiting similar or better scores on MT quality metrics. We demonstrate that this finding is borne out in human evaluations as well. We then show that these differences are especially pronounced when translating sentences that contain idiomatic expressions.