Sentence-State LSTM for Text Representation

TL;DR

This paper introduces Sentence-State LSTM, an alternative to traditional BiLSTMs, using parallel states for each word to improve text representation by enabling simultaneous local and global information exchange.

Contribution

The paper proposes a novel Sentence-State LSTM structure that overcomes limitations of sequential BiLSTMs by allowing parallel processing of word states for enhanced text encoding.

Findings

Achieves competitive performance on classification benchmarks

Demonstrates strong representation power with fewer parameters

Outperforms traditional stacked BiLSTM models

Abstract

Bi-directional LSTMs are a powerful tool for text representation. On the other hand, they have been shown to suffer various limitations due to their sequential nature. We investigate an alternative LSTM structure for encoding text, which consists of a parallel state for each word. Recurrent steps are used to perform local and global information exchange between words simultaneously, rather than incremental reading of a sequence of words. Results on various classification and sequence labelling benchmarks show that the proposed model has strong representation power, giving highly competitive performances compared to stacked BiLSTM models with similar parameter numbers.