What do the metrics mean? A critical analysis of the use of Automated Evaluation Metrics in Interpreting

TL;DR

This paper critically examines automated evaluation metrics for interpreting quality, arguing that current metrics lack the ability to account for communicative context and are insufficient alone for assessing interpreting quality.

Contribution

The article provides a critical analysis of recent automated quality measurement methods, highlighting their limitations in capturing authentic interpreting practice and contextual factors.

Findings

Automated metrics cannot fully account for communicative context.

Current metrics are inadequate for evaluating authentic interpreting quality.

Contextual factors are fundamental in interpreting quality assessment.

Abstract

With the growth of interpreting technologies, from remote interpreting and Computer-Aided Interpreting to automated speech translation and interpreting avatars, there is now a high demand for ways to quickly and efficiently measure the quality of any interpreting delivered. A range of approaches to fulfil the need for quick and efficient quality measurement have been proposed, each involving some measure of automation. This article examines these recently-proposed quality measurement methods and will discuss their suitability for measuring the quality of authentic interpreting practice, whether delivered by humans or machines, concluding that automatic metrics as currently proposed cannot take into account the communicative context and thus are not viable measures of the quality of any interpreting provision when used on their own. Across all attempts to measure or even categorise quality in Interpreting Studies, the contexts in which interpreting takes place have become fundamental to the final analysis.