Towards Generalising Neural Topical Representations

TL;DR

This paper proposes a novel framework that enhances neural topic models by minimizing semantic distance between similar documents using TopicalOT, significantly improving their ability to generalize across different corpora.

Contribution

The work introduces a plug-and-play module for NTMs that leverages text data augmentation and TopicalOT to improve cross-corpus generalization of topical representations.

Findings

Significant improvement in cross-corpus generalization of NTMs.

Framework is compatible with most existing NTMs.

Demonstrated effectiveness through extensive experiments.

Abstract

Topic models have evolved from conventional Bayesian probabilistic models to recent Neural Topic Models (NTMs). Although NTMs have shown promising performance when trained and tested on a specific corpus, their generalisation ability across corpora has yet to be studied. In practice, we often expect that an NTM trained on a source corpus can still produce quality topical representation (i.e., latent distribution over topics) for the document from different target corpora to a certain degree. In this work, we aim to improve NTMs further so that their representation power for documents generalises reliably across corpora and tasks. To do so, we propose to enhance NTMs by narrowing the semantic distance between similar documents, with the underlying assumption that documents from different corpora may share similar semantics. Specifically, we obtain a similar document for each training document by text data augmentation. Then, we optimise NTMs further by minimising the semantic distance between each pair, measured by the Topical Optimal Transport (TopicalOT) distance, which computes the optimal transport distance between their topical representations. Our framework can be readily applied to most NTMs as a plug-and-play module. Extensive experiments show that our framework significantly improves the generalisation ability regarding neural topical representation across corpora. Our code and datasets are available at: https://github.com/Xiaohao-Yang/Topic_Model_Generalisation.