Tied Multitask Learning for Neural Speech Translation

TL;DR

This paper introduces a tied multitask learning approach for neural speech translation that leverages decoder interconnections and regularization to improve low-resource transcription and translation, as well as word discovery.

Contribution

It proposes a novel tied multitask model with decoder connections and regularization techniques, enhancing speech translation and word discovery performance.

Findings

Improved low-resource speech transcription and translation accuracy.

Enhanced word discovery using attention over unsegmented input.

Better model performance with decoder interconnections and regularization.

Abstract

We explore multitask models for neural translation of speech, augmenting them in order to reflect two intuitive notions. First, we introduce a model where the second task decoder receives information from the decoder of the first task, since higher-level intermediate representations should provide useful information. Second, we apply regularization that encourages transitivity and invertibility. We show that the application of these notions on jointly trained models improves performance on the tasks of low-resource speech transcription and translation. It also leads to better performance when using attention information for word discovery over unsegmented input.