Stream-level Latency Evaluation for Simultaneous Machine Translation

TL;DR

This paper introduces a stream-level latency evaluation method for simultaneous machine translation, addressing the limitations of sentence-level measures by considering the continuous streaming context to better reflect real-world translation policies.

Contribution

It proposes a novel stream-level latency measure using re-segmentation, improving the accuracy of latency evaluation in streaming translation scenarios.

Findings

Stream-level latency measures better reflect real streaming conditions.

Re-segmentation approach improves latency evaluation coherence.

Method validated on IWSLT streaming translation task.

Abstract

Simultaneous machine translation has recently gained traction thanks to significant quality improvements and the advent of streaming applications. Simultaneous translation systems need to find a trade-off between translation quality and response time, and with this purpose multiple latency measures have been proposed. However, latency evaluations for simultaneous translation are estimated at the sentence level, not taking into account the sequential nature of a streaming scenario. Indeed, these sentence-level latency measures are not well suited for continuous stream translation resulting in figures that are not coherent with the simultaneous translation policy of the system being assessed. This work proposes a stream-level adaptation of the current latency measures based on a re-segmentation approach applied to the output translation, that is successfully evaluated on streaming conditions for a reference IWSLT task.