Recurrent Neural Networks with Mixed Hierarchical Structures for Natural Language Processing

TL;DR

This paper introduces MHS-RNN, a hierarchical RNN model with static and dynamic boundaries and attention mechanisms, designed to better capture linguistic structures for improved document classification.

Contribution

It proposes a novel multi-layer hierarchical RNN with mixed static and dynamic boundaries, enhanced by attention mechanisms, for more effective natural language understanding.

Findings

Outperforms previous methods on five datasets

Effectively captures hierarchical linguistic structures

Improves document classification accuracy

Abstract

Hierarchical structures exist in both linguistics and Natural Language Processing (NLP) tasks. How to design RNNs to learn hierarchical representations of natural languages remains a long-standing challenge. In this paper, we define two different types of boundaries referred to as static and dynamic boundaries, respectively, and then use them to construct a multi-layer hierarchical structure for document classification tasks. In particular, we focus on a three-layer hierarchical structure with static word- and sentence- layers and a dynamic phrase-layer. LSTM cells and two boundary detectors are used to implement the proposed structure, and the resulting network is called the {\em Recurrent Neural Network with Mixed Hierarchical Structures} (MHS-RNN). We further add three layers of attention mechanisms to the MHS-RNN model. Incorporating attention mechanisms allows our model to use more important content to construct document representation and enhance its performance on document classification tasks. Experiments on five different datasets show that the proposed architecture outperforms previous methods on all the five tasks.