SubER: A Metric for Automatic Evaluation of Subtitle Quality

TL;DR

SubER is a new comprehensive metric for automatically evaluating subtitle quality, considering transcription accuracy, segmentation, and timing, and it correlates well with human judgments and editing effort.

Contribution

It introduces SubER, a novel metric based on edit distance with shifts that jointly assesses subtitle transcription, segmentation, and timing quality.

Findings

SubER outperforms existing metrics like WER and BLEU in correlating with human assessments.

It shows high correlation with post-editing effort in subtitle correction.

The metric effectively integrates multiple aspects of subtitle quality.

Abstract

This paper addresses the problem of evaluating the quality of automatically generated subtitles, which includes not only the quality of the machine-transcribed or translated speech, but also the quality of line segmentation and subtitle timing. We propose SubER - a single novel metric based on edit distance with shifts that takes all of these subtitle properties into account. We compare it to existing metrics for evaluating transcription, translation, and subtitle quality. A careful human evaluation in a post-editing scenario shows that the new metric has a high correlation with the post-editing effort and direct human assessment scores, outperforming baseline metrics considering only the subtitle text, such as WER and BLEU, and existing methods to integrate segmentation and timing features.