A Formal Theory of Survey Experiment Generalizability: Attention and Salience

TL;DR

This paper develops a formal theory explaining how survey environments influence attention and salience, affecting the generalizability of experimental effects to real-world settings.

Contribution

It introduces a behavioral microfoundations framework to understand and improve the transportability of survey experiment results.

Findings

Consideration-set compression can amplify effects beyond real-world magnitudes.

Context-dependent salience can cause sign instability between survey and real-world effects.

The theory clarifies conditions under which survey effects generalize and how to improve survey design.

Abstract

Survey experiments are widely used to identify causal effects in political science and the social sciences. Yet researchers are typically interested in more than the internal validity of an experimentally induced contrast. They also want to know whether the estimated effect corresponds to the effect in the real world. We develop a formal theory of survey experiment generalizability grounded in behavioral microfoundations. The theory highlights two mechanisms. First, the survey environment shapes attention: it determines which considerations enter the respondent's active consideration set. Second, it shapes salience: conditional on consideration, it influences the relative weight assigned to those considerations. This framework yields two main results. Consideration-set compression generates amplification: survey-experimental effects can be larger in magnitude than their real-world counterparts, even for the same individuals, treatment content, and outcome. Context-dependent salience generates sign instability: the direction of the survey effect need not coincide with the direction of the corresponding real-world effect. The theory clarifies what survey experiments identify, when those effects are likely to generalize, and how survey designs can be modified to improve decision-environment transportability.