What do the metrics mean? A critical analysis of the use of Automated Evaluation Metrics in Interpreting
Jonathan Downie, Joss Moorkens

TL;DR
This paper critically examines automated evaluation metrics for interpreting quality, arguing that current metrics lack the ability to account for communicative context and are insufficient alone for assessing interpreting quality.
Contribution
The article provides a critical analysis of recent automated quality measurement methods, highlighting their limitations in capturing authentic interpreting practice and contextual factors.
Findings
Automated metrics cannot fully account for communicative context.
Current metrics are inadequate for evaluating authentic interpreting quality.
Contextual factors are fundamental in interpreting quality assessment.
Abstract
With the growth of interpreting technologies, from remote interpreting and Computer-Aided Interpreting to automated speech translation and interpreting avatars, there is now a high demand for ways to quickly and efficiently measure the quality of any interpreting delivered. A range of approaches to fulfil the need for quick and efficient quality measurement have been proposed, each involving some measure of automation. This article examines these recently-proposed quality measurement methods and will discuss their suitability for measuring the quality of authentic interpreting practice, whether delivered by humans or machines, concluding that automatic metrics as currently proposed cannot take into account the communicative context and thus are not viable measures of the quality of any interpreting provision when used on their own. Across all attempts to measure or even categorise…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsInterpreting and Communication in Healthcare · Translation Studies and Practices · Speech and dialogue systems
