Topic Modeling for Free-Response Text Data from a Complex Survey

TL;DR

This paper enhances topic modeling for complex survey data by incorporating survey weights into the Mixture of Unigrams model, enabling unbiased extraction of themes from open-ended survey responses.

Contribution

It introduces a pseudolikelihood approach for the Mixture of Unigrams model under informative sampling and develops a hierarchical version accounting for respondent-level factors.

Findings

The weighted MoU effectively extracts meaningful topics from survey data.

The hierarchical MoU captures variations in topic proportions across respondent characteristics.

Application to ANES data demonstrates improved interpretability of survey responses.

Abstract

Topic Modeling is a popular statistical tool commonly used on textual data to identify the hidden thematic structure in a document collection based on the distribution of words. Additionally, it can be used to cluster the documents, with clusters representing distinct topics. The Mixture of Unigrams (MoU) is a standard topic model for clustering document-term data and can be particularly useful for analyzing open-ended survey responses to extract meaningful information from the underlying topics. However, with complex survey designs, where data is often collected on individual (document) characteristics, it is essential to account for the sample design in order to avoid biased estimates. To address this issue, we propose the MoU model under informative sampling using a pseudolikelihood to account for the sample design in the model by incorporating survey weights. We evaluate the effectiveness of this approach through a simulation study and illustrate its application using two datasets from the American National Election Studies (ANES). We compare our pseudolikelihood-based MoU model to the traditional MoU and assess its effectiveness in extracting meaningful topics from survey data. Additionally, we introduce a hierarchical Mixture of Unigrams (hMoU) accounting for informative sampling where topic proportions are defined as functions of document-level fixed and random effects. We demonstrate the effectiveness of the proposed model through an application to ANES data comparing topic proportions across respondent-level factors such as gender, race, age group, and state.