Document Informed Neural Autoregressive Topic Models with Distributional Prior

TL;DR

This paper introduces neural autoregressive topic models that leverage full context and external knowledge via embeddings, significantly improving performance on both long and short texts in terms of coherence, generalization, and applicability.

Contribution

It extends neural autoregressive topic models to incorporate full context and external embeddings, enhancing their effectiveness on diverse text datasets.

Findings

Models outperform state-of-the-art in generalization and interpretability.

Effective on both long and short texts across multiple domains.

Improved retrieval and classification performance.

Abstract

We address two challenges in topic models: (1) Context information around words helps in determining their actual meaning, e.g., "networks" used in the contexts "artificial neural networks" vs. "biological neuron networks". Generative topic models infer topic-word distributions, taking no or only little context into account. Here, we extend a neural autoregressive topic model to exploit the full context information around words in a document in a language modeling fashion. The proposed model is named as iDocNADE. (2) Due to the small number of word occurrences (i.e., lack of context) in short text and data sparsity in a corpus of few documents, the application of topic models is challenging on such texts. Therefore, we propose a simple and efficient way of incorporating external knowledge into neural autoregressive topic models: we use embeddings as a distributional prior. The proposed variants are named as DocNADEe and iDocNADEe. We present novel neural autoregressive topic model variants that consistently outperform state-of-the-art generative topic models in terms of generalization, interpretability (topic coherence) and applicability (retrieval and classification) over 7 long-text and 8 short-text datasets from diverse domains.