Depth-Adaptive Graph Recurrent Network for Text Classification

TL;DR

This paper introduces a depth-adaptive graph recurrent network for text classification that dynamically allocates computational steps per word and incorporates sequential information, improving performance and efficiency.

Contribution

It proposes a novel depth-adaptive mechanism for S-LSTM that adjusts computation per word and integrates an RNN layer for sequential info, enhancing text classification.

Findings

Significant accuracy improvements over S-LSTM and Transformer.

Effective balance between model speed and accuracy.

Robust performance across 24 diverse datasets.

Abstract

The Sentence-State LSTM (S-LSTM) is a powerful and high efficient graph recurrent network, which views words as nodes and performs layer-wise recurrent steps between them simultaneously. Despite its successes on text representations, the S-LSTM still suffers from two drawbacks. Firstly, given a sentence, certain words are usually more ambiguous than others, and thus more computation steps need to be taken for these difficult words and vice versa. However, the S-LSTM takes fixed computation steps for all words, irrespective of their hardness. The secondary one comes from the lack of sequential information (e.g., word order) that is inherently important for natural language. In this paper, we try to address these issues and propose a depth-adaptive mechanism for the S-LSTM, which allows the model to learn how many computational steps to conduct for different words as required. In addition, we integrate an extra RNN layer to inject sequential information, which also serves as an input feature for the decision of adaptive depths. Results on the classic text classification task (24 datasets in various sizes and domains) show that our model brings significant improvements against the conventional S-LSTM and other high-performance models (e.g., the Transformer), meanwhile achieving a good accuracy-speed trade off.