Mixed Membership Word Embeddings for Computational Social Science

TL;DR

This paper introduces a probabilistic mixed membership word embedding method that enhances interpretability and performance in NLP tasks, especially suited for computational social science, with less data requirement.

Contribution

It presents a novel probabilistic model combining word embeddings and mixed membership modeling, enabling interpretable embeddings without large datasets.

Findings

Up to 63% improvement in language modeling accuracy

Embeddings are effective for supervised learning tasks

Models provide interpretable insights in social science case studies

Abstract

Word embeddings improve the performance of NLP systems by revealing the hidden structural relationships between words. Despite their success in many applications, word embeddings have seen very little use in computational social science NLP tasks, presumably due to their reliance on big data, and to a lack of interpretability. I propose a probabilistic model-based word embedding method which can recover interpretable embeddings, without big data. The key insight is to leverage mixed membership modeling, in which global representations are shared, but individual entities (i.e. dictionary words) are free to use these representations to uniquely differing degrees. I show how to train the model using a combination of state-of-the-art training techniques for word embeddings and topic models. The experimental results show an improvement in predictive language modeling of up to 63% in MRR over the skip-gram, and demonstrate that the representations are beneficial for supervised learning. I illustrate the interpretability of the models with computational social science case studies on State of the Union addresses and NIPS articles.