Interpretable probabilistic embeddings: bridging the gap between topic models and neural networks

TL;DR

This paper introduces probabilistic embeddings that combine topic models and neural word embeddings, achieving comparable performance to existing methods while enhancing interpretability and efficiency across multiple modalities.

Contribution

It merges probabilistic topic models with neural embeddings using an online EM algorithm, creating interpretable embeddings that outperform some existing models in various tasks.

Findings

Embeddings perform on par with SGNS in word similarity tasks.

Probabilistic document embeddings outperform paragraph2vec.

Multimodal embeddings improve inter-modality similarity and interpretability.

Abstract

We consider probabilistic topic models and more recent word embedding techniques from a perspective of learning hidden semantic representations. Inspired by a striking similarity of the two approaches, we merge them and learn probabilistic embeddings with online EM-algorithm on word co-occurrence data. The resulting embeddings perform on par with Skip-Gram Negative Sampling (SGNS) on word similarity tasks and benefit in the interpretability of the components. Next, we learn probabilistic document embeddings that outperform paragraph2vec on a document similarity task and require less memory and time for training. Finally, we employ multimodal Additive Regularization of Topic Models (ARTM) to obtain a high sparsity and learn embeddings for other modalities, such as timestamps and categories. We observe further improvement of word similarity performance and meaningful inter-modality similarities.