Neural Embedding Allocation: Distributed Representations of Topic Models

TL;DR

Neural Embedding Allocation (NEA) unifies topic models and embedding models into a single framework, producing interpretable vector representations that improve performance and coherence in large-topic scenarios.

Contribution

NEA is a novel algorithm that deconstructs topic models into neural embeddings, enhancing interpretability and performance across various models.

Findings

NEA outperforms state-of-the-art models in embedding quality.

Using NEA improves topic coherence scores.

NEA generalizes across different topic modeling frameworks.

Abstract

Word embedding models such as the skip-gram learn vector representations of words' semantic relationships, and document embedding models learn similar representations for documents. On the other hand, topic models provide latent representations of the documents' topical themes. To get the benefits of these representations simultaneously, we propose a unifying algorithm, called neural embedding allocation (NEA), which deconstructs topic models into interpretable vector-space embeddings of words, topics, documents, authors, and so on, by learning neural embeddings to mimic the topic models. We showcase NEA's effectiveness and generality on LDA, author-topic models and the recently proposed mixed membership skip gram topic model and achieve better performance with the embeddings compared to several state-of-the-art models. Furthermore, we demonstrate that using NEA to smooth out the topics improves coherence scores over the original topic models when the number of topics is large.