Contrastive Learning for Neural Topic Model

TL;DR

This paper introduces a novel neural topic modeling approach that incorporates relations among similar documents and external information, improving topic coherence across diverse datasets.

Contribution

It reformulates adversarial neural topic models as an optimization problem and designs a new sampling method to better integrate external variables and document relations.

Findings

Outperforms state-of-the-art models on benchmark datasets

Achieves higher topic coherence scores

Effectively incorporates external information into topic modeling

Abstract

Recent empirical studies show that adversarial topic models (ATM) can successfully capture semantic patterns of the document by differentiating a document with another dissimilar sample. However, utilizing that discriminative-generative architecture has two important drawbacks: (1) the architecture does not relate similar documents, which has the same document-word distribution of salient words; (2) it restricts the ability to integrate external information, such as sentiments of the document, which has been shown to benefit the training of neural topic model. To address those issues, we revisit the adversarial topic architecture in the viewpoint of mathematical analysis, propose a novel approach to re-formulate discriminative goal as an optimization problem, and design a novel sampling method which facilitates the integration of external variables. The reformulation encourages the model to incorporate the relations among similar samples and enforces the constraint on the similarity among dissimilar ones; while the sampling method, which is based on the internal input and reconstructed output, helps inform the model of salient words contributing to the main topic. Experimental results show that our framework outperforms other state-of-the-art neural topic models in three common benchmark datasets that belong to various domains, vocabulary sizes, and document lengths in terms of topic coherence.