Bridging the Gaps of Both Modality and Language: Synchronous Bilingual CTC for Speech Translation and Speech Recognition

TL;DR

This paper introduces a synchronous bilingual CTC framework that improves speech translation and recognition by leveraging dual objectives for cross-lingual and modality bridging, achieving state-of-the-art results especially in resource-limited settings.

Contribution

The paper proposes a novel synchronous bilingual CTC model with an enhanced variant, BiL-CTC+, that advances speech translation and recognition performance under resource constraints.

Findings

Achieves state-of-the-art results on MuST-C benchmarks.

Significantly improves speech recognition performance.

Demonstrates the effectiveness of cross-lingual learning in speech tasks.

Abstract

In this study, we present synchronous bilingual Connectionist Temporal Classification (CTC), an innovative framework that leverages dual CTC to bridge the gaps of both modality and language in the speech translation (ST) task. Utilizing transcript and translation as concurrent objectives for CTC, our model bridges the gap between audio and text as well as between source and target languages. Building upon the recent advances in CTC application, we develop an enhanced variant, BiL-CTC+, that establishes new state-of-the-art performances on the MuST-C ST benchmarks under resource-constrained scenarios. Intriguingly, our method also yields significant improvements in speech recognition performance, revealing the effect of cross-lingual learning on transcription and demonstrating its broad applicability. The source code is available at https://github.com/xuchennlp/S2T.