Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voice

TL;DR

Seed-LiveInterpret 2.0 is an end-to-end speech-to-speech translation system that offers high accuracy, ultra-low latency, and voice cloning, significantly outperforming existing solutions in quality and speed.

Contribution

The paper introduces a novel duplex framework for simultaneous speech translation with voice cloning, achieving better accuracy and latency than prior systems.

Findings

Achieves over 70% correctness in complex translation scenarios.

Reduces latency from 10 seconds to 3 seconds, a 70% improvement.

Outperforms commercial solutions in translation quality.

Abstract

Simultaneous Interpretation (SI) represents one of the most daunting frontiers in the translation industry, with product-level automatic systems long plagued by intractable challenges: subpar transcription and translation quality, lack of real-time speech generation, multi-speaker confusion, and translated speech inflation, especially in long-form discourses. In this study, we introduce Seed-LiveInterpret 2.0, an end-to-end SI model that delivers high-fidelity, ultra-low-latency speech-to-speech generation with voice cloning capabilities. As a fully operational product-level solution, Seed-LiveInterpret 2.0 tackles these challenges head-on through our novel duplex speech-to-speech understanding-generating framework. Experimental results demonstrate that through large-scale pretraining and reinforcement learning, the model achieves a significantly better balance between translation accuracy and latency, validated by human interpreters to exceed 70% correctness in complex scenarios. Notably, Seed-LiveInterpret 2.0 outperforms commercial SI solutions by significant margins in translation quality, while slashing the average latency of cloned speech from nearly 10 seconds to a near-real-time 3 seconds, which is around a near 70% reduction that drastically enhances practical usability.