TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency

TL;DR

TopicRNN combines recurrent neural networks with latent topic models to effectively capture both local syntactic and global semantic dependencies in documents, improving word prediction and enabling unsupervised feature extraction.

Contribution

It introduces an end-to-end trainable model that integrates RNNs and latent topics, capturing long-range semantic dependencies alongside local dependencies.

Findings

Outperforms existing contextual RNN baselines in word prediction

Achieves competitive sentiment analysis error rate of 6.28%

Generates sensible topics comparable to LDA

Abstract

In this paper, we propose TopicRNN, a recurrent neural network (RNN)-based language model designed to directly capture the global semantic meaning relating words in a document via latent topics. Because of their sequential nature, RNNs are good at capturing the local structure of a word sequence - both semantic and syntactic - but might face difficulty remembering long-range dependencies. Intuitively, these long-range dependencies are of semantic nature. In contrast, latent topic models are able to capture the global underlying semantic structure of a document but do not account for word ordering. The proposed TopicRNN model integrates the merits of RNNs and latent topic models: it captures local (syntactic) dependencies using an RNN and global (semantic) dependencies using latent topics. Unlike previous work on contextual RNN language modeling, our model is learned end-to-end. Empirical results on word prediction show that TopicRNN outperforms existing contextual RNN baselines. In addition, TopicRNN can be used as an unsupervised feature extractor for documents. We do this for sentiment analysis on the IMDB movie review dataset and report an error rate of $6.28\%$. This is comparable to the state-of-the-art $5.91\%$ resulting from a semi-supervised approach. Finally, TopicRNN also yields sensible topics, making it a useful alternative to document models such as latent Dirichlet allocation.