Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications

TL;DR

This paper proposes reframing sentence-level quality estimation of machine translation as a classification task, enabling the use of smaller, more efficient models suitable for real-world applications, instead of relying on large, computationally expensive models.

Contribution

It demonstrates that classification-based QE models can match state-of-the-art performance while being more efficient, challenging the necessity of large models for regression-based QE.

Findings

Model compression techniques perform poorly for QE regression.

Full model parameterization is needed for state-of-the-art results.

Reframing QE as classification better aligns with real-world application needs.

Abstract

Sentence-level Quality estimation (QE) of machine translation is traditionally formulated as a regression task, and the performance of QE models is typically measured by Pearson correlation with human labels. Recent QE models have achieved previously-unseen levels of correlation with human judgments, but they rely on large multilingual contextualized language models that are computationally expensive and make them infeasible for real-world applications. In this work, we evaluate several model compression techniques for QE and find that, despite their popularity in other NLP tasks, they lead to poor performance in this regression setting. We observe that a full model parameterization is required to achieve SoTA results in a regression task. However, we argue that the level of expressiveness of a model in a continuous range is unnecessary given the downstream applications of QE, and show that reframing QE as a classification problem and evaluating QE models using classification metrics would better reflect their actual performance in real-world applications.