Neural Dynamic Focused Topic Model

TL;DR

This paper introduces a neural dynamic focused topic model that tracks topic appearances over time, improving generalization and convergence speed compared to existing models, especially for temporally evolving documents.

Contribution

It proposes a neural approach using Bernoulli variables to decouple topic activity from proportions, enhancing dynamic topic modeling capabilities.

Findings

Outperforms state-of-the-art models in generalization tasks

Performs comparably on prediction tasks with similar parameters

Converges approximately twice as fast as existing models

Abstract

Topic models and all their variants analyse text by learning meaningful representations through word co-occurrences. As pointed out by Williamson et al. (2010), such models implicitly assume that the probability of a topic to be active and its proportion within each document are positively correlated. This correlation can be strongly detrimental in the case of documents created over time, simply because recent documents are likely better described by new and hence rare topics. In this work we leverage recent advances in neural variational inference and present an alternative neural approach to the dynamic Focused Topic Model. Indeed, we develop a neural model for topic evolution which exploits sequences of Bernoulli random variables in order to track the appearances of topics, thereby decoupling their activities from their proportions. We evaluate our model on three different datasets (the UN general debates, the collection of NeurIPS papers, and the ACL Anthology dataset) and show that it (i) outperforms state-of-the-art topic models in generalization tasks and (ii) performs comparably to them on prediction tasks, while employing roughly the same number of parameters, and converging about two times faster. Source code to reproduce our experiments is available online.