Recurrent Dropout without Memory Loss

TL;DR

This paper introduces a novel recurrent dropout method that drops neurons in recurrent connections without losing long-term memory, improving RNN performance on NLP tasks.

Contribution

It proposes a simple, effective recurrent dropout technique that preserves memory, enhancing RNN regularization without complex modifications.

Findings

Consistent performance improvements on NLP benchmarks

Effective when combined with standard dropout methods

Applicable to LSTM networks with easy implementation

Abstract

This paper presents a novel approach to recurrent neural network (RNN) regularization. Differently from the widely adopted dropout method, which is applied to \textit{forward} connections of feed-forward architectures or RNNs, we propose to drop neurons directly in \textit{recurrent} connections in a way that does not cause loss of long-term memory. Our approach is as easy to implement and apply as the regular feed-forward dropout and we demonstrate its effectiveness for Long Short-Term Memory network, the most popular type of RNN cells. Our experiments on NLP benchmarks show consistent improvements even when combined with conventional feed-forward dropout.