Variational Gaussian Topic Model with Invertible Neural Projections

TL;DR

This paper introduces VaGTM and VaGTM-IP, neural topic models that incorporate word relatedness via Gaussian distributions and invertible neural projections, leading to more coherent topics.

Contribution

It proposes a novel variational auto-encoder based topic model that integrates word embeddings with Gaussian assumptions and invertible neural projections.

Findings

VaGTM and VaGTM-IP outperform baselines in coherence.

Models effectively incorporate word relatedness.

Experiments verify improved topic quality.

Abstract

Neural topic models have triggered a surge of interest in extracting topics from text automatically since they avoid the sophisticated derivations in conventional topic models. However, scarce neural topic models incorporate the word relatedness information captured in word embedding into the modeling process. To address this issue, we propose a novel topic modeling approach, called Variational Gaussian Topic Model (VaGTM). Based on the variational auto-encoder, the proposed VaGTM models each topic with a multivariate Gaussian in decoder to incorporate word relatedness. Furthermore, to address the limitation that pre-trained word embeddings of topic-associated words do not follow a multivariate Gaussian, Variational Gaussian Topic Model with Invertible neural Projections (VaGTM-IP) is extended from VaGTM. Three benchmark text corpora are used in experiments to verify the effectiveness of VaGTM and VaGTM-IP. The experimental results show that VaGTM and VaGTM-IP outperform several competitive baselines and obtain more coherent topics.