Cyber Risks of Machine Translation Critical Errors : Arabic Mental Health Tweets as a Case Study

TL;DR

This paper presents a dataset of critical machine translation errors in Arabic mental health tweets, highlighting safety issues and the inadequacy of current quality metrics in detecting such errors.

Contribution

It introduces an annotated dataset of Arabic mental health tweet mistranslations and analyzes the failure of existing metrics to identify critical errors.

Findings

Current metrics do not penalize critical errors

NMT hallucinations can produce fluent but harmful mistranslations

The dataset exposes safety and ethical risks in machine translation

Abstract

With the advent of Neural Machine Translation (NMT) systems, the MT output has reached unprecedented accuracy levels which resulted in the ubiquity of MT tools on almost all online platforms with multilingual content. However, NMT systems, like other state-of-the-art AI generative systems, are prone to errors that are deemed machine hallucinations. The problem with NMT hallucinations is that they are remarkably \textit{fluent} hallucinations. Since they are trained to produce grammatically correct utterances, NMT systems are capable of producing mistranslations that are too fluent to be recognised by both users of the MT tool, as well as by automatic quality metrics that are used to gauge their performance. In this paper, we introduce an authentic dataset of machine translation critical errors to point to the ethical and safety issues involved in the common use of MT. The dataset comprises mistranslations of Arabic mental health postings manually annotated with critical error types. We also show how the commonly used quality metrics do not penalise critical errors and highlight this as a critical issue that merits further attention from researchers.