The Proxy Presumption: From Semantic Embeddings to Valid Social Measures

TL;DR

This paper addresses the challenge of validating social measures derived from semantic embeddings, proposing a protocol and methods to ensure their scientific validity in social science research.

Contribution

It introduces the Construct Validity Protocol and Counterfactual Neutralization to improve the validity of social constructs measured via embeddings.

Findings

The CVP provides a systematic pipeline for validation.

Counterfactual Neutralization reduces confounding in embeddings.

A Validity Suite tests various aspects of construct validity.

Abstract

Natural Language Processing is rapidly evolving into a primary instrument for Computational Social Science, with researchers increasingly using embeddings to measure latent constructs such as novelty, creativity, and bias. However, this transition faces a fundamental validity challenge: the ''Proxy Presumption,'' or the reliance on geometric properties (e.g., cosine distance) as direct measures of social concepts. We argue that without explicit validation, unsupervised representations remain entangled mixtures of the target construct ($C$) and confounding attributes ($Z$) like topic, style, and authorship. To bridge the gap between semantic embeddings and valid social measures, we introduce the Construct Validity Protocol (CVP). Drawing on causal representation learning and psychometrics, the CVP offers a rigorous pipeline from conceptualization to quantitative verification. We further propose Counterfactual Neutralization, a novel method using LLMs to reduce confounding in embedding space. By providing a standardized Validity Suite -- including tests for discriminant, incremental, and predictive validity -- this work offers the community a toolkit to transform heuristic proxies into robust, scientifically defensible instruments.