FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation

TL;DR

FRMT introduces a new dataset and benchmark for evaluating few-shot region-aware machine translation, focusing on style-targeted translation between English and regional variants of Portuguese and Mandarin Chinese.

Contribution

It provides a novel dataset, evaluation metrics, and baseline models for region-aware machine translation, facilitating research in style and regional variation translation.

Findings

Automatic metrics correlate well with human judgments.

Baseline models demonstrate the feasibility of region-aware translation.

The dataset enables detailed analysis of regional translation phenomena.

Abstract

We present FRMT, a new dataset and evaluation benchmark for Few-shot Region-aware Machine Translation, a type of style-targeted translation. The dataset consists of professional translations from English into two regional variants each of Portuguese and Mandarin Chinese. Source documents are selected to enable detailed analysis of phenomena of interest, including lexically distinct terms and distractor terms. We explore automatic evaluation metrics for FRMT and validate their correlation with expert human evaluation across both region-matched and mismatched rating scenarios. Finally, we present a number of baseline models for this task, and offer guidelines for how researchers can train, evaluate, and compare their own models. Our dataset and evaluation code are publicly available: https://bit.ly/frmt-task