Densely Connected Bidirectional LSTM with Applications to Sentence Classification

TL;DR

This paper introduces a densely connected bidirectional LSTM model that effectively addresses vanishing-gradient issues and improves sentence classification performance across multiple benchmarks.

Contribution

It proposes a novel multi-layer RNN architecture with dense connections, enabling deeper networks and better performance in NLP tasks.

Findings

Successfully trained models with up to 20 layers.

Significant improvements over traditional Bi-LSTM.

Competitive performance compared to state-of-the-art methods.

Abstract

Deep neural networks have recently been shown to achieve highly competitive performance in many computer vision tasks due to their abilities of exploring in a much larger hypothesis space. However, since most deep architectures like stacked RNNs tend to suffer from the vanishing-gradient and overfitting problems, their effects are still understudied in many NLP tasks. Inspired by this, we propose a novel multi-layer RNN model called densely connected bidirectional long short-term memory (DC-Bi-LSTM) in this paper, which essentially represents each layer by the concatenation of its hidden state and all preceding layers' hidden states, followed by recursively passing each layer's representation to all subsequent layers. We evaluate our proposed model on five benchmark datasets of sentence classification. DC-Bi-LSTM with depth up to 20 can be successfully trained and obtain significant improvements over the traditional Bi-LSTM with the same or even less parameters. Moreover, our model has promising performance compared with the state-of-the-art approaches.