In your own words: computationally identifying interpretable themes in free-text survey data

TL;DR

This paper introduces a computational framework called 'In Your Own Words' for identifying interpretable themes in free-text survey responses, enhancing systematic analysis and revealing nuanced insights.

Contribution

The paper presents a novel method that produces more coherent themes from free-text data, aiding survey design and understanding heterogeneity in responses.

Findings

Themes are more coherent and interpretable than previous methods.

Identified salient constructs like belonging and identity fluidity.

Revealed heterogeneity within categories and discordance between self and perceived identities.

Abstract

Free-text survey responses can provide nuance often missed by structured questions, but remain difficult to statistically analyze. To address this, we introduce In Your Own Words, a computational framework for exploratory analyses of free-text survey data that identifies structured, interpretable themes in free-text responses, facilitating systematic analysis. To illustrate the benefits of this approach, we apply it to a new dataset of free-text descriptions of race, gender, and sexual orientation from 1,004 U.S. participants. The themes our approach produces on this dataset are more coherent and interpretable than those produced by past computational methods. The themes have three practical applications in survey research. First, they can suggest structured questions to add to future surveys by surfacing salient constructs - such as belonging and identity fluidity - that existing surveys do not capture. Second, the themes reveal heterogeneity within standardized categories, explaining additional variation in health, well-being, and identity importance. Third, the themes illuminate systematic discordance between self-identified and perceived identities, highlighting mechanisms of misrecognition that existing measures do not reflect. More broadly, our framework can be deployed in a wide range of survey settings to identify interpretable themes from free text, complementing existing qualitative methods.