Regularizing Recurrent Networks - On Injected Noise and Norm-based Methods

TL;DR

This paper evaluates various regularization techniques for RNNs, including noise injection and norm-based methods, and finds that noise training does not enhance RNN performance as previously conjectured.

Contribution

The study provides an empirical comparison of regularization methods on RNNs, highlighting the limited effectiveness of noise injection for improving long-term dependency learning.

Findings

Noise injection does not improve RNN performance.

Norm-based regularizers show limited benefits.

Advanced regularization methods like fast-dropout outperform simpler techniques.

Abstract

Advancements in parallel processing have lead to a surge in multilayer perceptrons' (MLP) applications and deep learning in the past decades. Recurrent Neural Networks (RNNs) give additional representational power to feedforward MLPs by providing a way to treat sequential data. However, RNNs are hard to train using conventional error backpropagation methods because of the difficulty in relating inputs over many time-steps. Regularization approaches from MLP sphere, like dropout and noisy weight training, have been insufficiently applied and tested on simple RNNs. Moreover, solutions have been proposed to improve convergence in RNNs but not enough to improve the long term dependency remembering capabilities thereof. In this study, we aim to empirically evaluate the remembering and generalization ability of RNNs on polyphonic musical datasets. The models are trained with injected noise, random dropout, norm-based regularizers and their respective performances compared to well-initialized plain RNNs and advanced regularization methods like fast-dropout. We conclude with evidence that training with noise does not improve performance as conjectured by a few works in RNN optimization before ours.