Alleviating Distribution Shift in Synthetic Data for Machine Translation Quality Estimation

TL;DR

This paper introduces DCSQE, a framework that reduces distribution shift in synthetic data for machine translation quality estimation, improving model performance by enhancing translation diversity and label quality.

Contribution

DCSQE employs constrained beam search and diverse models to generate more aligned synthetic data, guiding both translation and annotation processes for better quality estimation.

Findings

DCSQE outperforms SOTA baselines like CometKiwi.

The framework improves both supervised and unsupervised QE tasks.

Enhanced synthetic data quality leads to better model performance.

Abstract

Quality Estimation (QE) models evaluate the quality of machine translations without reference translations, serving as the reward models for the translation task. Due to the data scarcity, synthetic data generation has emerged as a promising solution. However, synthetic QE data often suffers from distribution shift, which can manifest as discrepancies between pseudo and real translations, or in pseudo labels that do not align with human preferences. To tackle this issue, we introduce DCSQE, a novel framework for alleviating distribution shift in synthetic QE data. To reduce the difference between pseudo and real translations, we employ the constrained beam search algorithm and enhance translation diversity through the use of distinct generation models. DCSQE uses references, i.e., translation supervision signals, to guide both the generation and annotation processes, enhancing the quality of token-level labels. DCSQE further identifies the shortest phrase covering consecutive error tokens, mimicking human annotation behavior, to assign the final phrase-level labels. Specially, we underscore that the translation model can not annotate translations of itself accurately. Extensive experiments demonstrate that DCSQE outperforms SOTA baselines like CometKiwi in both supervised and unsupervised settings. Further analysis offers insights into synthetic data generation that could benefit reward models for other tasks. The code is available at https://github.com/NJUNLP/njuqe.