TTA: Transcribe, Translate and Alignment for Cross-lingual Speech Representation

TL;DR

This paper introduces TTA, a lightweight model for cross-lingual speech representation that outperforms Whisper in various speech understanding tasks by leveraging large-scale multilingual training.

Contribution

The paper presents a novel TTA model optimized for speech semantics, trained on 358k hours of multilingual data, enhancing cross-lingual speech understanding and integration with LLMs.

Findings

TTA surpasses Whisper in ASR and speech translation benchmarks.

TTA demonstrates robust cross-lingual speech representations.

Training on large-scale data improves speech-text alignment performance.

Abstract

Speech-LLM models have demonstrated great performance in multi-modal and multi-task speech understanding. A typical speech-LLM paradigm is integrating speech modality with a large language model (LLM). While the Whisper encoder was frequently adopted in previous studies for speech input, it shows limitations regarding input format, model scale, and semantic performance. To this end, we propose a lightweight TTA model specialized in speech semantics for more effective LLM integration. With large-scale training of 358k hours of speech data on multilingual speech recognition (ASR), speech translation (ST) and speech-text alignment tasks, TTA is capable of producing robust cross-lingual speech representations. Extensive evaluations across diverse benchmarks, including ASR/ST, speech retrieval, and ASR-LLM performance assessments, demonstrate TTA's superiority over Whisper. Furthermore, we rigorously validate the interplay between cross-lingual capabilities and ASR/ST performance. The model weights and training recipes of TTA will be released as part of an audio understanding toolkit Auden.