The Prediction-Measurement Gap: Toward Meaning Representations as Scientific Instruments

TL;DR

This paper addresses the gap between predictive performance and scientific usability in text embeddings, proposing new objectives and evaluation methods to improve their interpretability and traceability as scientific instruments.

Contribution

It introduces a framework emphasizing geometric interpretability, robustness, and traceability for meaning representations, grounded in cognitive science, and proposes design and evaluation strategies.

Findings

Static embeddings are more transparent for measurement.

Contextual embeddings offer richer semantics but pose interpretability challenges.

Proposed methods aim to improve geometric interpretability and robustness.

Abstract

Text embeddings have become central to computational social science and psychology, enabling scalable measurement of meaning and mixed-method inference. Yet most representation learning is optimized and evaluated for prediction and retrieval, yielding a prediction-measurement gap: representations that perform well as features may be poorly suited as scientific instruments. The paper argues that scientific meaning analysis motivates a distinct family of objectives - scientific usability - emphasizing geometric legibility, interpretability and traceability to linguistic evidence, robustness to non-semantic confounds, and compatibility with regression-style inference over semantic directions. Grounded in cognitive and neuro-psychological views of meaning, the paper assesses static word embeddings and contextual transformer representations against these requirements: static spaces remain attractive for transparent measurement, whereas contextual spaces offer richer semantics but entangle meaning with other signals and exhibit geometric and interpretability issues that complicate inference. The paper then outlines a course-setting agenda around (i) geometry-first design for gradients and abstraction, including hierarchy-aware spaces constrained by psychologically privileged levels; (ii) invertible post-hoc transformations that recondition embedding geometry and reduce nuisance influence; and (iii) meaning atlases and measurement-oriented evaluation protocols for reliable and traceable semantic inference. As the field debates the limits of scale-first progress, measurement-ready representations offer a principled new frontier.