Comparison and Adaptation of Automatic Evaluation Metrics for Quality Assessment of Re-Speaking

TL;DR

This paper evaluates various automatic metrics for assessing re-speaking quality, comparing them to human judgments to identify effective methods for automatic quality estimation in live broadcast subtitles.

Contribution

It systematically compares multiple automatic evaluation metrics against human assessments in re-speaking, highlighting their relative effectiveness and adaptation potential.

Findings

BLEU and METEOR show strong correlation with human judgments

Automatic metrics can approximate human quality assessments

Some metrics outperform others in specific re-speaking contexts

Abstract

Re-speaking is a mechanism for obtaining high quality subtitles for use in live broadcast and other public events. Because it relies on humans performing the actual re-speaking, the task of estimating the quality of the results is non-trivial. Most organisations rely on humans to perform the actual quality assessment, but purely automatic methods have been developed for other similar problems, like Machine Translation. This paper will try to compare several of these methods: BLEU, EBLEU, NIST, METEOR, METEOR-PL, TER and RIBES. These will then be matched to the human-derived NER metric, commonly used in re-speaking.