Sensitivity Analysis for Survey Weights

TL;DR

This paper introduces two sensitivity analysis methods to assess the impact of unobserved or unmeasured confounders on survey weights, helping researchers evaluate bias in survey results, demonstrated through U.S. election polls.

Contribution

It proposes novel sensitivity analysis techniques for unobserved confounders in survey weighting, including graphical, numerical summaries, and benchmarking tools.

Findings

Graphical and numerical summaries of potential bias

Benchmarking approach for sensitivity assessment

Application to 2020 U.S. Presidential Election polls

Abstract

Survey weighting allows researchers to account for bias in survey samples, due to unit nonresponse or convenience sampling, using measured demographic covariates. Unfortunately, in practice, it is impossible to know whether the estimated survey weights are sufficient to alleviate concerns about bias due to unobserved confounders or incorrect functional forms used in weighting. In the following paper, we propose two sensitivity analyses for the exclusion of important covariates: (1) a sensitivity analysis for partially observed confounders (i.e., variables measured across the survey sample, but not the target population), and (2) a sensitivity analysis for fully unobserved confounders (i.e., variables not measured in either the survey or the target population). We provide graphical and numerical summaries of the potential bias that arises from such confounders, and introduce a benchmarking approach that allows researchers to quantitatively reason about the sensitivity of their results. We demonstrate our proposed sensitivity analyses using state-level 2020 U.S. Presidential Election polls.