Using Total Margin of Error to Account for Non-Sampling Error in Election Polls: The Case of Nonresponse

TL;DR

This paper proposes using total margin of error (TME), based on mean square error, to better account for both sampling and non-sampling errors, especially nonresponse, in election polls.

Contribution

It introduces a method to measure and incorporate total margin of error in poll reporting, including nonresponse bias, and compares new estimates to conventional ones.

Findings

TME measurement captures non-sampling errors effectively.

New estimates reduce potential bias from nonresponse.

Method improves poll accuracy by considering total error sources.

Abstract

The potential impact of non-sampling errors on election polls is well known, but measurement has focused on the margin of sampling error. Survey statisticians have long recommended measurement of total survey error by mean square error (MSE), which jointly measures sampling and non-sampling errors. We think it reasonable to use the square root of maximum MSE to measure the total margin of error (TME). Measurement of TME should encompass both sampling error and all forms of non-sampling error. We suggest that measurement of TME should be a standard feature in the reporting of polls. To provide a clear illustration, and because we believe the exceedingly low response rates commonly obtained by election polls to be a particularly worrisome source of potential error, we demonstrate how to measure the potential impact of nonresponse using the concept of TME. We first show how to measure TME when a pollster lacks any knowledge of the candidate preferences of nonrespondents. We then extend the analysis to settings where the pollster has partial knowledge that bounds the preferences of non-respondents. In each setting, we derive a simple poll estimate that approximately minimizes TME, a midpoint estimate, and compare it to a conventional poll estimate.