Evaluating the Construct Validity of Text Embeddings with Application to Survey Questions

TL;DR

This paper evaluates the construct validity of various text embedding models for social science applications, demonstrating their potential and limitations in representing survey questions and predicting responses.

Contribution

It adapts the construct validity framework to high-dimensional text embeddings and compares multiple embedding methods for survey question analysis.

Findings

BERT and Universal Sentence Encoder show higher validity in representing survey questions.

Embeddings can predict responses to new survey questions.

Construct validity varies across embedding methods.

Abstract

Text embedding models from Natural Language Processing can map text data (e.g. words, sentences, documents) to supposedly meaningful numerical representations (a.k.a. text embeddings). While such models are increasingly applied in social science research, one important issue is often not addressed: the extent to which these embeddings are valid representations of constructs relevant for social science research. We therefore propose the use of the classic construct validity framework to evaluate the validity of text embeddings. We show how this framework can be adapted to the opaque and high-dimensional nature of text embeddings, with application to survey questions. We include several popular text embedding methods (e.g. fastText, GloVe, BERT, Sentence-BERT, Universal Sentence Encoder) in our construct validity analyses. We find evidence of convergent and discriminant validity in some cases. We also show that embeddings can be used to predict respondent's answers to completely new survey questions. Furthermore, BERT-based embedding techniques and the Universal Sentence Encoder provide more valid representations of survey questions than do others. Our results thus highlight the necessity to examine the construct validity of text embeddings before deploying them in social science research.