Document Informed Neural Autoregressive Topic Models

TL;DR

This paper introduces a neural autoregressive topic model that leverages full contextual information around words, leading to improved performance, interpretability, and applicability across diverse datasets.

Contribution

It extends existing neural autoregressive topic models by incorporating full context information, enhancing generalization and interpretability in topic modeling.

Findings

Outperforms state-of-the-art generative topic models across seven datasets.

Achieves 9.6% improvement in precision at retrieval fraction 0.02.

Gains 7.2% in F1 score for text categorization.

Abstract

Context information around words helps in determining their actual meaning, for example "networks" used in contexts of artificial neural networks or biological neuron networks. Generative topic models infer topic-word distributions, taking no or only little context into account. Here, we extend a neural autoregressive topic model to exploit the full context information around words in a document in a language modeling fashion. This results in an improved performance in terms of generalization, interpretability and applicability. We apply our modeling approach to seven data sets from various domains and demonstrate that our approach consistently outperforms stateof-the-art generative topic models. With the learned representations, we show on an average a gain of 9.6% (0.57 Vs 0.52) in precision at retrieval fraction 0.02 and 7.2% (0.582 Vs 0.543) in F1 for text categorization.