CA*: Addressing Evaluation Pitfalls in Computation-Aware Latency for Simultaneous Speech Translation

TL;DR

This paper identifies flaws in current latency evaluation methods for Simultaneous Speech Translation, revealing misconceptions and proposing a corrected metric to better measure real-world latency performance.

Contribution

It uncovers fundamental misconceptions in existing latency metrics and introduces a modified approach for more accurate computation-aware latency measurement in SimulST.

Findings

Existing metrics overestimate latency in streaming settings

The root cause is a fundamental misconception in current evaluation methods

Proposed metric improves accuracy of latency measurement

Abstract

Simultaneous speech translation (SimulST) systems must balance translation quality with response time, making latency measurement crucial for evaluating their real-world performance. However, there has been a longstanding belief that current metrics yield unrealistically high latency measurements in unsegmented streaming settings. In this paper, we investigate this phenomenon, revealing its root cause in a fundamental misconception underlying existing latency evaluation approaches. We demonstrate that this issue affects not only streaming but also segment-level latency evaluation across different metrics. Furthermore, we propose a modification to correctly measure computation-aware latency for SimulST systems, addressing the limitations present in existing metrics.