Leap-LSTM: Enhancing Long Short-Term Memory for Text Categorization

TL;DR

Leap-LSTM is a novel model that improves text categorization by dynamically skipping irrelevant words, leading to faster processing and better accuracy compared to standard LSTM and previous skip models.

Contribution

The paper introduces Leap-LSTM, a dynamic skipping mechanism for LSTM that enhances efficiency and performance in long text processing tasks.

Findings

Leap-LSTM reads faster than standard LSTM.

Leap-LSTM achieves higher accuracy on multiple datasets.

Leap-LSTM offers better performance-efficiency trade-offs.

Abstract

Recurrent Neural Networks (RNNs) are widely used in the field of natural language processing (NLP), ranging from text categorization to question answering and machine translation. However, RNNs generally read the whole text from beginning to end or vice versa sometimes, which makes it inefficient to process long texts. When reading a long document for a categorization task, such as topic categorization, large quantities of words are irrelevant and can be skipped. To this end, we propose Leap-LSTM, an LSTM-enhanced model which dynamically leaps between words while reading texts. At each step, we utilize several feature encoders to extract messages from preceding texts, following texts and the current word, and then determine whether to skip the current word. We evaluate Leap-LSTM on several text categorization tasks: sentiment analysis, news categorization, ontology classification and topic classification, with five benchmark data sets. The experimental results show that our model reads faster and predicts better than standard LSTM. Compared to previous models which can also skip words, our model achieves better trade-offs between performance and efficiency.