On the Reliability of Multiple Systems Estimation for the Quantification of Modern Slavery

TL;DR

This paper critically evaluates the use of multiple systems estimation (MSE) for quantifying modern slavery, addressing controversies by analyzing datasets, assessing estimator bias, and proposing new reliability visualization methods.

Contribution

It provides a comprehensive review of MSE assumptions, introduces a reproducible analysis framework, and proposes novel methods to assess and visualize MSE reliability in modern slavery estimates.

Findings

MSE assumptions often lead to biased estimates when misspecified.

Internal consistency checks reveal variability in MSE accuracy.

New visualization techniques improve understanding of estimate robustness.

Abstract

The quantification of modern slavery has received increased attention recently as organizations have come together to produce global estimates, where multiple systems estimation (MSE) is often used to this end. Echoing a long-standing controversy, disagreements have re-surfaced regarding the underlying MSE assumptions, the robustness of MSE methodology, and the accuracy of MSE estimates in this application. Our goal is to help address and move past these controversies. To do so, we review MSE, its assumptions, and commonly used models for modern slavery applications. We introduce all of the publicly available modern slavery datasets in the literature, providing a reproducible analysis and highlighting current issues. Specifically, we utilize an internal consistency approach that constructs subsets of data for which ground truth is available, allowing us to evaluate the accuracy of MSE estimators. Next, we propose a characterization of the large sample bias of estimators as a function of misspecified assumptions. Then, we propose an alternative to traditional (e.g., bootstrap-based) assessments of reliability, which allows us to visualize trajectories of MSE estimates to illustrate the robustness of estimates. Finally, our complementary analyses are used to provide guidance regarding the application and reliability of MSE methodology.