xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metrics

TL;DR

This paper develops a compressed, efficient version of the xCOMET machine translation evaluation metric using distillation and quantization, maintaining high quality while reducing computational costs.

Contribution

It introduces xCOMET-lite, a significantly smaller and faster metric that retains most of the original's evaluation quality through novel compression techniques.

Findings

Quantization compresses xCOMET up to three times without quality loss.

Distillation produces xCOMET-lite with only 2.6% of parameters but 92.1% of original quality.

xCOMET-lite outperforms smaller existing metrics on the WMT22 dataset.

Abstract

State-of-the-art trainable machine translation evaluation metrics like xCOMET achieve high correlation with human judgment but rely on large encoders (up to 10.7B parameters), making them computationally expensive and inaccessible to researchers with limited resources. To address this issue, we investigate whether the knowledge stored in these large encoders can be compressed while maintaining quality. We employ distillation, quantization, and pruning techniques to create efficient xCOMET alternatives and introduce a novel data collection pipeline for efficient black-box distillation. Our experiments show that, using quantization, xCOMET can be compressed up to three times with no quality degradation. Additionally, through distillation, we create an 278M-sized xCOMET-lite metric, which has only 2.6% of xCOMET-XXL parameters, but retains 92.1% of its quality. Besides, it surpasses strong small-scale metrics like COMET-22 and BLEURT-20 on the WMT22 metrics challenge dataset by 6.4%, despite using 50% fewer parameters. All code, dataset, and models are available online at https://github.com/NL2G/xCOMET-lite.