IsoChronoMeter: A simple and effective isochronic translation evaluation metric

TL;DR

IsoChronoMeter is a new metric for evaluating the timing accuracy of translations, especially useful for video dubbing, that is simple, scalable, and does not require gold standard data, leveraging TTS duration predictors.

Contribution

The paper introduces IsoChronoMeter, a novel isochronic translation evaluation metric that is resource-efficient and effective for assessing translation timing in automatic dubbing.

Findings

ICM reveals shortcomings of current translation systems

ICM correlates well with human judgments of isochrony

The metric enables scalable evaluation without gold data

Abstract

Machine translation (MT) has come a long way and is readily employed in production systems to serve millions of users daily. With the recent advances in generative AI, a new form of translation is becoming possible - video dubbing. This work motivates the importance of isochronic translation, especially in the context of automatic dubbing, and introduces `IsoChronoMeter' (ICM). ICM is a simple yet effective metric to measure isochrony of translations in a scalable and resource-efficient way without the need for gold data, based on state-of-the-art text-to-speech (TTS) duration predictors. We motivate IsoChronoMeter and demonstrate its effectiveness. Using ICM we demonstrate the shortcomings of state-of-the-art translation systems and show the need for new methods. We release the code at this URL: \url{https://github.com/braskai/isochronometer}.