Killing Two Flies with One Stone: An Attempt to Break LLMs Using English->Icelandic Idioms and Proper Names

TL;DR

This paper evaluates the challenges faced by translation models in accurately translating English idioms and proper names into Icelandic, revealing significant room for improvement in handling these complex linguistic features.

Contribution

The paper introduces two test suites for assessing translation quality of idioms and proper names in English to Icelandic translation models, highlighting their current limitations.

Findings

Low translation accuracy for idiomatic expressions

Poor handling of Icelandic exonyms and inflected proper names

Indicates substantial room for improvement in translation models

Abstract

This paper presents the submission of the \'Arni Magn\'usson Institute's team to the WMT24 test suite subtask, focusing on idiomatic expressions and proper names for the English->Icelandic translation direction. Intuitively and empirically, idioms and proper names are known to be a significant challenge for modern translation models. We create two different test suites. The first evaluates the competency of MT systems in translating common English idiomatic expressions, as well as testing whether systems can distinguish between those expressions and the same phrases when used in a literal context. The second test suite consists of place names that should be translated into their Icelandic exonyms (and correctly inflected) and pairs of Icelandic names that share a surface form between the male and female variants, so that incorrect translations impact meaning as well as readability. The scores reported are relatively low, especially for idiomatic expressions and place names, and indicate considerable room for improvement.