Thinking Slow about Latency Evaluation for Simultaneous Machine Translation

TL;DR

This paper introduces Differentiable Average Lagging (DAL), a new latency metric for simultaneous machine translation that is differentiable and mathematically consistent, improving latency evaluation for real-time translation systems.

Contribution

The paper proposes DAL, a novel latency metric for simultaneous translation that enhances interpretability and mathematical consistency over existing metrics.

Findings

DAL is differentiable, enabling optimization-based improvements.

DAL provides more accurate latency measurement in real-time translation.

Experimental results show DAL's effectiveness over previous metrics.

Abstract

Simultaneous machine translation attempts to translate a source sentence before it is finished being spoken, with applications to translation of spoken language for live streaming and conversation. Since simultaneous systems trade quality to reduce latency, having an effective and interpretable latency metric is crucial. We introduce a variant of the recently proposed Average Lagging (AL) metric, which we call Differentiable Average Lagging (DAL). It distinguishes itself by being differentiable and internally consistent to its underlying mathematical model.