Neural Topic Modeling with Continual Lifelong Learning

TL;DR

This paper introduces a lifelong neural topic modeling framework that continuously learns from document streams, transferring knowledge to improve topic coherence and retrieval in sparse data scenarios.

Contribution

It proposes a novel lifelong learning approach for neural topic modeling that mitigates data sparsity issues through knowledge transfer and prevents catastrophic forgetting.

Findings

Improved perplexity scores on sparse document collections.

Enhanced topic coherence and quality.

Better performance in information retrieval tasks.

Abstract

Lifelong learning has recently attracted attention in building machine learning systems that continually accumulate and transfer knowledge to help future learning. Unsupervised topic modeling has been popularly used to discover topics from document collections. However, the application of topic modeling is challenging due to data sparsity, e.g., in a small collection of (short) documents and thus, generate incoherent topics and sub-optimal document representations. To address the problem, we propose a lifelong learning framework for neural topic modeling that can continuously process streams of document collections, accumulate topics and guide future topic modeling tasks by knowledge transfer from several sources to better deal with the sparse data. In the lifelong process, we particularly investigate jointly: (1) sharing generative homologies (latent topics) over lifetime to transfer prior knowledge, and (2) minimizing catastrophic forgetting to retain the past learning via novel selective data augmentation, co-training and topic regularization approaches. Given a stream of document collections, we apply the proposed Lifelong Neural Topic Modeling (LNTM) framework in modeling three sparse document collections as future tasks and demonstrate improved performance quantified by perplexity, topic coherence and information retrieval task.