Lifelong Neural Topic Learning in Contextualized Autoregressive Topic Models of Language via Informative Transfers

TL;DR

This paper introduces a neural lifelong learning framework for topic models that incorporates linguistic structures and mitigates catastrophic forgetting, enabling continuous learning from sequential document collections.

Contribution

It proposes a contextualized neural topic model with a lifelong learning mechanism and data augmentation to improve topic coherence and knowledge retention.

Findings

Enhanced topic coherence through linguistic features.

Effective lifelong learning with minimal catastrophic forgetting.

Reduced need for complete historical data during training.

Abstract

Topic models such as LDA, DocNADE, iDocNADEe have been popular in document analysis. However, the traditional topic models have several limitations including: (1) Bag-of-words (BoW) assumption, where they ignore word ordering, (2) Data sparsity, where the application of topic models is challenging due to limited word co-occurrences, leading to incoherent topics and (3) No Continuous Learning framework for topic learning in lifelong fashion, exploiting historical knowledge (or latent topics) and minimizing catastrophic forgetting. This thesis focuses on addressing the above challenges within neural topic modeling framework. We propose: (1) Contextualized topic model that combines a topic and a language model and introduces linguistic structures (such as word ordering, syntactic and semantic features, etc.) in topic modeling, (2) A novel lifelong learning mechanism into neural topic modeling framework to demonstrate continuous learning in sequential document collections and minimizing catastrophic forgetting. Additionally, we perform a selective data augmentation to alleviate the need for complete historical corpora during data hallucination or replay.