Accelerating recurrent neural network training using sequence bucketing and multi-GPU data parallelization

TL;DR

This paper introduces a method to accelerate recurrent neural network training by combining sequence bucketing based on input length with multi-GPU data parallelization, significantly improving training efficiency for variable-length sequences.

Contribution

The paper presents a novel combination of sequence bucketing and multi-GPU data parallelization to enhance RNN training speed, especially for variable-length input sequences.

Findings

Sequence bucketing reduces training time compared to baseline.

Multi-GPU parallelization further accelerates training.

Effective for online handwriting recognition with LSTM RNNs.

Abstract

An efficient algorithm for recurrent neural network training is presented. The approach increases the training speed for tasks where a length of the input sequence may vary significantly. The proposed approach is based on the optimal batch bucketing by input sequence length and data parallelization on multiple graphical processing units. The baseline training performance without sequence bucketing is compared with the proposed solution for a different number of buckets. An example is given for the online handwriting recognition task using an LSTM recurrent neural network. The evaluation is performed in terms of the wall clock time, number of epochs, and validation loss value.