CODET: A Benchmark for Contrastive Dialectal Evaluation of Machine Translation

TL;DR

This paper introduces CODET, a benchmark with 891 dialectal variations across twelve languages, to evaluate and improve the robustness of machine translation systems against dialectal differences.

Contribution

The paper presents a new benchmark dataset, CODET, for assessing dialectal robustness in machine translation, addressing a gap in existing evaluation resources.

Findings

Large MT models struggle with dialectal variations.

CODET reveals significant performance degradation on dialectal data.

Benchmark facilitates targeted improvements in dialectal translation robustness.

Abstract

Neural machine translation (NMT) systems exhibit limited robustness in handling source-side linguistic variations. Their performance tends to degrade when faced with even slight deviations in language usage, such as different domains or variations introduced by second-language speakers. It is intuitive to extend this observation to encompass dialectal variations as well, but the work allowing the community to evaluate MT systems on this dimension is limited. To alleviate this issue, we compile and release CODET, a contrastive dialectal benchmark encompassing 891 different variations from twelve different languages. We also quantitatively demonstrate the challenges large MT models face in effectively translating dialectal variants. All the data and code have been released.