RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition

TL;DR

RETURNN is a flexible neural toolkit that enables fast training and decoding of attention models, achieving state-of-the-art results in translation and speech recognition through efficient CUDA kernels and a versatile architecture.

Contribution

The paper introduces RETURNN, a general neural toolkit that combines fast CUDA LSTM kernels with a flexible architecture, facilitating rapid experimentation and state-of-the-art performance in translation and speech recognition.

Findings

Fast training and decoding speeds for attention models.

Layer-wise pretraining improves BLEU scores and enables deeper networks.

Achieved state-of-the-art results on WMT 2017 and Switchboard datasets.

Abstract

We compare the fast training and decoding speed of RETURNN of attention models for translation, due to fast CUDA LSTM kernels, and a fast pure TensorFlow beam search decoder. We show that a layer-wise pretraining scheme for recurrent attention models gives over 1% BLEU improvement absolute and it allows to train deeper recurrent encoder networks. Promising preliminary results on max. expected BLEU training are presented. We are able to train state-of-the-art models for translation and end-to-end models for speech recognition and show results on WMT 2017 and Switchboard. The flexibility of RETURNN allows a fast research feedback loop to experiment with alternative architectures, and its generality allows to use it on a wide range of applications.