Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation

TL;DR

This paper introduces MT-Pref, a large dataset of 18,000 translation preferences derived from human assessments and automatic metrics, to improve machine translation quality by aligning models with human-like preferences.

Contribution

It presents a novel method for creating a preference dataset by combining human judgments with automatic metrics, enhancing translation model training.

Findings

Aligning models on MT-Pref improves translation quality on benchmarks.

The dataset covers 18 languages and multiple domains.

Automatic metrics can effectively recover human preferences.

Abstract

Alignment with human preferences is an important step in developing accurate and safe large language models. This is no exception in machine translation (MT), where better handling of language nuances and context-specific variations leads to improved quality. However, preference data based on human feedback can be very expensive to obtain and curate at a large scale. Automatic metrics, on the other hand, can induce preferences, but they might not match human expectations perfectly. In this paper, we propose an approach that leverages the best of both worlds. We first collect sentence-level quality assessments from professional linguists on translations generated by multiple high-quality MT systems and evaluate the ability of current automatic metrics to recover these preferences. We then use this analysis to curate a new dataset, MT-Pref (metric induced translation preference) dataset, which comprises 18k instances covering 18 language directions, using texts sourced from multiple domains post-2022. We show that aligning TOWER models on MT-Pref significantly improves translation quality on WMT23 and FLORES benchmarks.