Health Estimate Differences Between Six Independent Web Surveys: Different Web Surveys, Different Results?

TL;DR

This study compares health estimate results across six independent web surveys from different agencies, revealing that survey methodology impacts health data accuracy despite calibration efforts.

Contribution

It systematically analyzes how different panel selection and management strategies influence health survey estimates, highlighting the limitations of weighting in reducing differences.

Findings

Differences exist between probability and non-probability surveys in health estimates.

Weighting reduces but does not eliminate survey estimate differences.

Non-probability surveys show larger discrepancies than expected even after calibration.

Abstract

Most general population web surveys are based on online panels maintained by commercial survey agencies. However, survey agencies differ in their panel selection and management strategies. Little is known if these different strategies cause differences in survey estimates. This paper presents the results of a systematic study designed to analyze the differences in web survey results between agencies. Six different survey agencies were commissioned with the same web survey using an identical standardized questionnaire covering factual health items. Five surveys were fielded at the same time. A calibration approach was used to control the effect of demographics on the outcome. Overall, the results show differences between probability and non-probability surveys in health estimates, which were reduced but not eliminated by weighting. Furthermore, the differences between non-probability surveys before and after weighting are larger than expected between random samples from the same population.