Deep Temporal-Recurrent-Replicated-Softmax for Topical Trends over Time

TL;DR

This paper introduces RNNRSM, an unsupervised neural dynamic topic model that captures temporal dependencies in document collections, outperforming existing models in trend detection, topic coherence, and word evolution analysis over 19 years of NLP research articles.

Contribution

The paper presents RNNRSM, a novel neural model that explicitly models temporal dependencies in topic evolution, improving over prior models in trend analysis and interpretability.

Findings

RNNRSM outperforms state-of-the-art models in generalization and topic coherence.

The model effectively captures the evolution of topics and words over time.

A new metric, SPAN, quantifies the model's ability to track word changes in topics.

Abstract

Dynamic topic modeling facilitates the identification of topical trends over time in temporal collections of unstructured documents. We introduce a novel unsupervised neural dynamic topic model named as Recurrent Neural Network-Replicated Softmax Model (RNNRSM), where the discovered topics at each time influence the topic discovery in the subsequent time steps. We account for the temporal ordering of documents by explicitly modeling a joint distribution of latent topical dependencies over time, using distributional estimators with temporal recurrent connections. Applying RNN-RSM to 19 years of articles on NLP research, we demonstrate that compared to state-of-the art topic models, RNNRSM shows better generalization, topic interpretation, evolution and trends. We also introduce a metric (named as SPAN) to quantify the capability of dynamic topic model to capture word evolution in topics over time.