StreamUni: Achieving Streaming Speech Translation with a Unified Large Speech-Language Model

TL;DR

StreamUni introduces a unified large speech-language model that performs streaming speech translation by integrating segmentation, policy, and translation tasks, achieving state-of-the-art results without extensive task-specific training.

Contribution

The paper presents StreamUni, a novel unified model that combines multiple stages of streaming speech translation into a single framework using speech Chain-of-Thought, reducing reliance on segmentation models and extensive training.

Findings

Achieves state-of-the-art performance on StreamST benchmarks.

Effectively balances low latency with high translation quality.

Reduces need for task-specific policy training.

Abstract

Streaming speech translation (StreamST) requires determining appropriate timing, known as policy, to generate translations while continuously receiving source speech inputs, balancing low latency with high translation quality. However, existing StreamST methods typically operate on sentence-level speech segments, referred to as simultaneous speech translation (SimulST). In practice, they require collaboration with segmentation models to accomplish StreamST, where the truncated speech segments constrain SimulST models to make policy decisions and generate translations based on limited contextual information. Moreover, SimulST models struggle to learn effective policies due to the complexity of speech inputs and cross-lingual generation. To address these challenges, we propose StreamUni, which achieves StreamST through a unified Large Speech-Language Model (LSLM). Specifically, StreamUni incorporates speech Chain-of-Thought (CoT) in guiding the LSLM to generate multi-stage outputs. Leveraging these multi-stage outputs, StreamUni simultaneously accomplishes speech segmentation, policy decision, and translation generation, completing StreamST without requiring massive policy-specific training. Additionally, we propose a streaming CoT training method that enhances low-latency policy decisions and generation capabilities using limited CoT data. Experiments demonstrate that our approach achieves state-of-the-art performance on StreamST tasks.