A comprehensive study of batch construction strategies for recurrent neural networks in MXNet

TL;DR

This paper compares batch construction strategies for training recurrent neural networks, proposing a simple alternating sorting method that matches the performance of more complex bucketing approaches in speech recognition tasks.

Contribution

It introduces a straightforward alternating sorting strategy for batch construction in RNN training, offering comparable results to bucketing with simpler implementation.

Findings

Alternating sorting matches bucketing in training time.

The method achieves similar recognition performance.

The approach is easier to implement.

Abstract

In this work we compare different batch construction methods for mini-batch training of recurrent neural networks. While popular implementations like TensorFlow and MXNet suggest a bucketing approach to improve the parallelization capabilities of the recurrent training process, we propose a simple ordering strategy that arranges the training sequences in a stochastic alternatingly sorted way. We compare our method to sequence bucketing as well as various other batch construction strategies on the CHiME-4 noisy speech recognition corpus. The experiments show that our alternated sorting approach is able to compete both in training time and recognition performance while being conceptually simpler to implement.