Benchmarking of LSTM Networks

TL;DR

This paper benchmarks LSTM networks using MNIST and UW3 datasets, analyzing how various architectural and hyperparameter choices affect performance, and identifies key factors influencing their effectiveness.

Contribution

It provides a systematic evaluation of LSTM configurations and hyperparameters, highlighting the impact of training methods and architectural features on performance.

Findings

Performance depends smoothly on learning rates

Batching and momentum have no significant effect

Softmax training outperforms least squares

Abstract

LSTM (Long Short-Term Memory) recurrent neural networks have been highly successful in a number of application areas. This technical report describes the use of the MNIST and UW3 databases for benchmarking LSTM networks and explores the effect of different architectural and hyperparameter choices on performance. Significant findings include: (1) LSTM performance depends smoothly on learning rates, (2) batching and momentum has no significant effect on performance, (3) softmax training outperforms least square training, (4) peephole units are not useful, (5) the standard non-linearities (tanh and sigmoid) perform best, (6) bidirectional training combined with CTC performs better than other methods.