Continual Neural Topic Model

TL;DR

The paper introduces CoNTM, a neural continual learning approach for topic modeling that updates topics over time without forgetting, outperforming existing dynamic models in quality and adaptability.

Contribution

It proposes a novel neural continual learning framework for topic models that maintains long-term memory and adapts online, filling a gap in existing dynamic and online models.

Findings

CoNTM outperforms dynamic topic models in topic quality.

CoNTM achieves lower predictive perplexity.

CoNTM captures more diverse and temporally relevant topics.

Abstract

In continual learning, our aim is to learn a new task without forgetting what was learned previously. In topic models, this translates to learning new topic models without forgetting previously learned topics. Previous work either considered Dynamic Topic Models (DTMs), which learn the evolution of topics based on the entire training corpus at once, or Online Topic Models, which are updated continuously based on new data but do not have long-term memory. To fill this gap, we propose the Continual Neural Topic Model (CoNTM), which continuously learns topic models at subsequent time steps without forgetting what was previously learned. This is achieved using a global prior distribution that is continuously updated. In our experiments, CoNTM consistently outperformed the dynamic topic model in terms of topic quality and predictive perplexity while being able to capture topic changes online. The analysis reveals that CoNTM can learn more diverse topics and better capture temporal changes than existing methods.