Efficient Transformer for Direct Speech Translation

TL;DR

This paper introduces an efficient Transformer-based model for direct speech translation that processes spectrograms without prior convolutional layers, maintaining information and achieving competitive results.

Contribution

The paper proposes using the Longformer as an encoder in speech translation, eliminating the need for convolutional layers before the Transformer.

Findings

Results are close to standard approaches.

The method preserves spectrogram information.

It offers a promising research direction.

Abstract

The advent of Transformer-based models has surpassed the barriers of text. When working with speech, we must face a problem: the sequence length of an audio input is not suitable for the Transformer. To bypass this problem, a usual approach is adding strided convolutional layers, to reduce the sequence length before using the Transformer. In this paper, we propose a new approach for direct Speech Translation, where thanks to an efficient Transformer we can work with a spectrogram without having to use convolutional layers before the Transformer. This allows the encoder to learn directly from the spectrogram and no information is lost. We have created an encoder-decoder model, where the encoder is an efficient Transformer -- the Longformer -- and the decoder is a traditional Transformer decoder. Our results, which are close to the ones obtained with the standard approach, show that this is a promising research direction.