Multilevel calibration weighting for survey data

TL;DR

This paper introduces multilevel calibration weighting combined with outcome modeling to improve survey bias correction, especially for complex interactions, demonstrated through re-analysis of 2016 U.S. election polls.

Contribution

It proposes a novel multilevel calibration weighting method that balances marginal and interaction constraints, enhancing survey adjustment techniques.

Findings

Improved accuracy in voter intention estimates for 2016 election.

Demonstrated benefits of the method over traditional raking and post-stratification.

Method implemented in the multical R package.

Abstract

In the November 2016 U.S. presidential election, many state level public opinion polls, particularly in the Upper Midwest, incorrectly predicted the winning candidate. One leading explanation for this polling miss is that the precipitous decline in traditional polling response rates led to greater reliance on statistical methods to adjust for the corresponding bias -- and that these methods failed to adjust for important interactions between key variables like education, race, and geographic region. Finding calibration weights that account for important interactions remains challenging with traditional survey methods: raking typically balances the margins alone, while post-stratification, which exactly balances all interactions, is only feasible for a small number of variables. In this paper, we propose multilevel calibration weighting, which enforces tight balance constraints for marginal balance and looser constraints for higher-order interactions. This incorporates some of the benefits of post-stratification while retaining the guarantees of raking. We then correct for the bias due to the relaxed constraints via a flexible outcome model; we call this approach Double Regression with Post-stratification (DRP). We characterize the asymptotic properties of these estimators and show that the proposed calibration approach has a dual representation as a multilevel model for survey response. We then use these tools to to re-assess a large-scale survey of voter intention in the 2016 U.S. presidential election, finding meaningful gains from the proposed methods. The approach is available in the multical R package.