Knowledge-Prompted Estimator: A Novel Approach to Explainable Machine Translation Assessment

TL;DR

This paper introduces the Knowledge-Prompted Estimator (KPE), a Chain-of-Thought prompting method that combines multiple techniques to improve segment-level and token-level quality estimation in machine translation, enhancing interpretability and performance.

Contribution

The paper presents KPE, a novel CoT prompting approach that integrates perplexity and similarity metrics for better MT quality estimation and interpretability at segment and token levels.

Findings

KPE outperforms previous models in segment-level estimation.

KPE significantly improves token alignment accuracy.

Enhanced interpretability for MT quality assessment.

Abstract

Cross-lingual Machine Translation (MT) quality estimation plays a crucial role in evaluating translation performance. GEMBA, the first MT quality assessment metric based on Large Language Models (LLMs), employs one-step prompting to achieve state-of-the-art (SOTA) in system-level MT quality estimation; however, it lacks segment-level analysis. In contrast, Chain-of-Thought (CoT) prompting outperforms one-step prompting by offering improved reasoning and explainability. In this paper, we introduce Knowledge-Prompted Estimator (KPE), a CoT prompting method that combines three one-step prompting techniques, including perplexity, token-level similarity, and sentence-level similarity. This method attains enhanced performance for segment-level estimation compared with previous deep learning models and one-step prompting approaches. Furthermore, supplementary experiments on word-level visualized alignment demonstrate that our KPE method significantly improves token alignment compared with earlier models and provides better interpretability for MT quality estimation. Code will be released upon publication.