DUB: Discrete Unit Back-translation for Speech Translation

TL;DR

This paper introduces Discrete Unit Back-translation (DUB), a novel method that uses discrete speech units to improve speech translation by effectively applying machine translation techniques, resulting in significant BLEU score improvements.

Contribution

The paper proposes DUB, a new back-translation approach using discrete speech units, demonstrating its effectiveness in enhancing speech translation performance.

Findings

DUB achieves an average 5.5 BLEU boost on MuST-C datasets.

Discrete units outperform continuous features in direct speech translation.

Method performs well in low-resource language scenarios.

Abstract

How can speech-to-text translation (ST) perform as well as machine translation (MT)? The key point is to bridge the modality gap between speech and text so that useful MT techniques can be applied to ST. Recently, the approach of representing speech with unsupervised discrete units yields a new way to ease the modality problem. This motivates us to propose Discrete Unit Back-translation (DUB) to answer two questions: (1) Is it better to represent speech with discrete units than with continuous features in direct ST? (2) How much benefit can useful MT techniques bring to ST? With DUB, the back-translation technique can successfully be applied on direct ST and obtains an average boost of 5.5 BLEU on MuST-C En-De/Fr/Es. In the low-resource language scenario, our method achieves comparable performance to existing methods that rely on large-scale external data. Code and models are available at https://github.com/0nutation/DUB.