Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq

TL;DR

OpenSeq2Seq is a TensorFlow toolkit enabling efficient mixed-precision training for sequence-to-sequence models, achieving state-of-the-art results in NLP and speech recognition with significantly reduced training time.

Contribution

It introduces a versatile, distributed, mixed-precision training toolkit that accelerates training and improves performance across various sequence-to-sequence tasks.

Findings

Achieves 1.5-3x faster training times.

Provides state-of-the-art performance on translation and speech recognition.

Supports diverse sequence-to-sequence applications.

Abstract

We present OpenSeq2Seq - a TensorFlow-based toolkit for training sequence-to-sequence models that features distributed and mixed-precision training. Benchmarks on machine translation and speech recognition tasks show that models built using OpenSeq2Seq give state-of-the-art performance at 1.5-3x less training time. OpenSeq2Seq currently provides building blocks for models that solve a wide range of tasks including neural machine translation, automatic speech recognition, and speech synthesis.