Unified Segment-to-Segment Framework for Simultaneous Sequence Generation

TL;DR

This paper introduces a unified framework for simultaneous sequence generation that adaptively learns optimal source-target mappings, improving performance and generality across tasks like streaming speech recognition and machine translation.

Contribution

The proposed Seg2Seg framework is the first to unify various simultaneous sequence generation tasks with an adaptive, learnable mapping approach, surpassing task-specific heuristics.

Findings

Achieves state-of-the-art results on multiple tasks

Demonstrates better generality across different sequence generation tasks

Effectively learns optimal generation moments through expectation training

Abstract

Simultaneous sequence generation is a pivotal task for real-time scenarios, such as streaming speech recognition, simultaneous machine translation and simultaneous speech translation, where the target sequence is generated while receiving the source sequence. The crux of achieving high-quality generation with low latency lies in identifying the optimal moments for generating, accomplished by learning a mapping between the source and target sequences. However, existing methods often rely on task-specific heuristics for different sequence types, limiting the model's capacity to adaptively learn the source-target mapping and hindering the exploration of multi-task learning for various simultaneous tasks. In this paper, we propose a unified segment-to-segment framework (Seg2Seg) for simultaneous sequence generation, which learns the mapping in an adaptive and unified manner. During the process of simultaneous generation, the model alternates between waiting for a source segment and generating a target segment, making the segment serve as the natural bridge between the source and target. To accomplish this, Seg2Seg introduces a latent segment as the pivot between source to target and explores all potential source-target mappings via the proposed expectation training, thereby learning the optimal moments for generating. Experiments on multiple simultaneous generation tasks demonstrate that Seg2Seg achieves state-of-the-art performance and exhibits better generality across various tasks.