Evaluating MT Systems: A Theoretical Framework

TL;DR

This paper proposes a theoretical framework for evaluating machine translation systems based on the concept of cognitive ease, which integrates adequacy and fluency, enabling modular and comprehensive assessment of various MT methods.

Contribution

It introduces a novel framework centered on cognitive ease for evaluating MT, allowing modular assessment across linguistic levels and applicability to new MT types.

Findings

Framework unifies existing evaluation metrics

Modular approach enables detailed analysis

Applicable to speech and discourse translation

Abstract

This paper outlines a theoretical framework using which different automatic metrics can be designed for evaluation of Machine Translation systems. It introduces the concept of {\em cognitive ease} which depends on {\em adequacy} and {\em lack of fluency}. Thus, cognitive ease becomes the main parameter to be measured rather than comprehensibility. The framework allows the components of cognitive ease to be broken up and computed based on different linguistic levels etc. Independence of dimensions and linearly combining them provides for a highly modular approach. The paper places the existing automatic methods in an overall framework, to understand them better and to improve upon them in future. It can also be used to evaluate the newer types of MT systems, such as speech to speech translation and discourse translation.