Is Human Annotation Necessary? Iterative MBR Distillation for Error Span Detection in Machine Translation

TL;DR

This paper introduces an iterative MBR distillation method that uses large language models to generate pseudo-labels for Error Span Detection in machine translation, reducing reliance on costly human annotations and outperforming supervised baselines.

Contribution

The paper presents a novel self-evolution framework using MBR decoding with LLM-generated pseudo-labels for ESD, eliminating the need for human annotations.

Findings

Models trained on pseudo-labels outperform supervised baselines.

The approach maintains competitive sentence-level performance.

Effective in reducing annotation costs and improving ESD accuracy.

Abstract

Error Span Detection (ESD) is a crucial subtask in Machine Translation (MT) evaluation, aiming to identify the location and severity of translation errors. While fine-tuning models on human-annotated data improves ESD performance, acquiring such data is expensive and prone to inconsistencies among annotators. To address this, we propose a novel self-evolution framework based on Minimum Bayes Risk (MBR) decoding, named Iterative MBR Distillation for ESD, which eliminates the reliance on human annotations by leveraging an off-the-shelf LLM to generate pseudo-labels. Extensive experiments on the WMT Metrics Shared Task datasets demonstrate that models trained solely on these self-generated pseudo-labels outperform both unadapted base model and supervised baselines trained on human annotations at the system and span levels, while maintaining competitive sentence-level performance.