Zero-Shot Translation Quality Estimation with Explicit Cross-Lingual Patterns

TL;DR

This paper introduces a zero-shot quality estimation model for translation that uses explicit cross-lingual patterns to address mismatching issues in BERTScore, achieving performance comparable or superior to supervised methods.

Contribution

The paper proposes a novel zero-shot QE approach incorporating explicit cross-lingual patterns, improving translation quality estimation without requiring labeled data.

Findings

Reduces mismatching errors in BERTScore-based QE

Achieves comparable performance to supervised QE methods

Outperforms supervised methods in 2 out of 6 translation directions

Abstract

This paper describes our submission of the WMT 2020 Shared Task on Sentence Level Direct Assessment, Quality Estimation (QE). In this study, we empirically reveal the \textit{mismatching issue} when directly adopting BERTScore to QE. Specifically, there exist lots of mismatching errors between the source sentence and translated candidate sentence with token pairwise similarity. In response to this issue, we propose to expose explicit cross-lingual patterns, \textit{e.g.} word alignments and generation score, to our proposed zero-shot models. Experiments show that our proposed QE model with explicit cross-lingual patterns could alleviate the mismatching issue, thereby improving the performance. Encouragingly, our zero-shot QE method could achieve comparable performance with supervised QE method, and even outperforms the supervised counterpart on 2 out of 6 directions. We expect our work could shed light on the zero-shot QE model improvement.