A new approach to calculating BERTScore for automatic assessment of translation quality

TL;DR

This paper proposes modifications to the BERTScore metric, including embedding alignment and token aggregation, to improve automatic translation quality assessment between English and Russian, achieving better correlation with human judgments.

Contribution

It introduces a novel approach to align monolingual embeddings and refine token matching in BERTScore, enhancing its effectiveness for sentence-level translation quality evaluation.

Findings

Improved correlation with human judgments using proposed modifications.

Embedding alignment reduces mismatching issues.

Enhanced token aggregation leads to better assessment accuracy.

Abstract

The study of the applicability of the BERTScore metric was conducted to translation quality assessment at the sentence level for English -> Russian direction. Experiments were performed with a pre-trained Multilingual BERT as well as with a pair of Monolingual BERT models. To align monolingual embeddings, an orthogonal transformation based on anchor tokens was used. It was demonstrated that such transformation helps to prevent mismatching issue and shown that this approach gives better results than using embeddings of the Multilingual model. To improve the token matching process it is proposed to combine all incomplete WorkPiece tokens into meaningful words and use simple averaging of corresponding vectors and to calculate BERTScore based on anchor tokens only. Such modifications allowed us to achieve a better correlation of the model predictions with human judgments. In addition to evaluating machine translation, several versions of human translation were evaluated as well, the problems of this approach were listed.