Measuring the Unmeasurable? Systematic Evidence on Scale Transformations in Subjective Survey Data

TL;DR

This paper systematically examines whether the common assumption that survey scale labels are linearly interpreted by respondents holds true, revealing that while signs are robust, effect size estimates are sensitive to scale non-linearities.

Contribution

It develops a framework to assess the impact of scale non-linearity on empirical results and provides experimental evidence showing mild deviations from linearity in survey responses.

Findings

Signs of effects are robust despite non-linearity.

Effect size estimates are unreliable under scale non-linearity.

Respondents deviate mildly from linear scale use.

Abstract

Ordered response scales are ubiquitous in economics, but their interpretation rests on an untested assumption: that numerical labels reflect equal psychological intervals. The contribution of this paper is to provide a systematic assessment of this linearity assumption, developing a general framework to quantify how easily empirical results can be overturned when it is relaxed. Using original experimental data, we show that respondents use survey scales in ways that deviate from linearity, but only mildly so. Focusing on wellbeing research, we then replicate 40,000+ coefficient estimates across more than 80 papers published in top economics journals. Coefficient signs are remarkably robust to the mild departures from linear scale-use we document experimentally. However, estimates of relative effect sizes, which are crucial for policy applications, are unreliable even under these modest non-linearities.