Incorporate Semantic Structures into Machine Translation Evaluation via UCCA

TL;DR

This paper introduces SWSS, a new machine translation evaluation method that uses UCCA to identify semantic core words, improving the accuracy of sentence similarity assessments by focusing on important semantic content.

Contribution

The paper proposes a novel MT evaluation approach that leverages UCCA to identify semantic core words and enhances existing metrics by emphasizing semantic importance.

Findings

SWSS improves performance of existing MT evaluation metrics.

Using semantic core words leads to more accurate translation quality assessment.

Experimental results validate the effectiveness of SWSS across multiple datasets.

Abstract

Copying mechanism has been commonly used in neural paraphrasing networks and other text generation tasks, in which some important words in the input sequence are preserved in the output sequence. Similarly, in machine translation, we notice that there are certain words or phrases appearing in all good translations of one source text, and these words tend to convey important semantic information. Therefore, in this work, we define words carrying important semantic meanings in sentences as semantic core words. Moreover, we propose an MT evaluation approach named Semantically Weighted Sentence Similarity (SWSS). It leverages the power of UCCA to identify semantic core words, and then calculates sentence similarity scores on the overlap of semantic core words. Experimental results show that SWSS can consistently improve the performance of popular MT evaluation metrics which are based on lexical similarity.