Improving Metrics for Speech Translation

TL;DR

This paper presents Parallel Paraphrasing, an augmentation technique that improves speech translation metrics by using automatic paraphrasing of references and hypotheses, leading to better correlation with human quality judgments.

Contribution

The paper introduces a novel paraphrasing-based augmentation method for translation metrics and new datasets for evaluating speech translation quality in Swiss German.

Findings

Improved correlation of metrics with human judgments using the proposed method.

Created two new datasets for Swiss German speech-to-text evaluation.

Demonstrated significant performance gains over standard metrics.

Abstract

We introduce Parallel Paraphrasing ($\text{Para}_\text{both}$), an augmentation method for translation metrics making use of automatic paraphrasing of both the reference and hypothesis. This method counteracts the typically misleading results of speech translation metrics such as WER, CER, and BLEU if only a single reference is available. We introduce two new datasets explicitly created to measure the quality of metrics intended to be applied to Swiss German speech-to-text systems. Based on these datasets, we show that we are able to significantly improve the correlation with human quality perception if our method is applied to commonly used metrics.