When Sensitivity Bias Varies Across Subgroups: The Impact of Non-uniform Polarity in List Experiments

TL;DR

This paper investigates how non-uniform sensitivity bias across subgroups affects the accuracy of list experiments, revealing that common estimators can be biased and diagnostics may produce false positives, especially when bias polarity varies.

Contribution

It introduces the concept of non-uniform polarity in sensitivity bias and systematically analyzes its impact on list experiment estimators through simulations.

Findings

Bias worsens with opposite polarity across groups

Standard estimators often produce biased prevalence estimates

Diagnostic tests can yield false positives

Abstract

Survey researchers face the problem of sensitivity bias: since people are reluctant to reveal socially undesirable or otherwise risky traits, aggregate estimates of these traits will be biased. List experiments offer a solution by conferring respondents greater privacy. However, little is know about how list experiments fare when sensitivity bias varies across respondent subgroups. For example, a trait that is socially undesirable to one group may socially desirable in a second group, leading sensitivity bias to be negative in the first group, while it is positive in the second. Or a trait may be not sensitive in one group, leading sensitivity bias to be zero in one group and non-zero in another. We use Monte Carlo simulations to explore what happens when the polarity (sign) of sensitivity bias is non-uniform. We find that a general diagnostic test yields false positives and that commonly used estimators return biased estimates of the prevalence of the sensitive trait, coefficients of covariates, and sensitivity bias itself. The bias is worse when polarity runs in opposite directions across subgroups, and as the difference in subgroup sizes increases. Significantly, non-uniform polarity could explain why some list experiments appear to 'fail'. By defining and systematically investigating the problem of non-uniform polarity, we hope to save some studies from the file-drawer and provide some guidance for future research.