Shining a Light on Forensic Black-Box Studies

TL;DR

This paper critically examines forensic black-box studies, revealing sampling biases and missing data issues that lead to underestimated error rates, and proposes solutions to improve their reliability for judicial use.

Contribution

It identifies key methodological flaws in black-box studies, such as non-representative sampling and non-ignorable missingness, and suggests concrete steps to address these issues.

Findings

Black-box studies use non-representative samples of examiners.

Ignoring missing data leads to underestimation of error rates.

Proposed steps can improve the accuracy of forensic error estimates.

Abstract

Forensic science plays a critical role in the United States criminal justice system. For decades, many feature-based fields of forensic science, such as firearm and toolmark identification, developed outside the scientific community's purview. The results of these studies are widely relied on by judges nationwide. However, this reliance is misplaced. Black-box studies to date suffer from inappropriate sampling methods and high rates of missingness. Current black-box studies ignore both problems in arriving at the error rate estimates presented to courts. We explore the impact of each type of limitation using available data from black-box studies and court materials. We show that black-box studies rely on non-representative samples of examiners. Using a case study of a popular ballistics study, we find evidence that these unrepresentative samples may commit fewer errors than the wider population from which they came. We also find evidence that the missingness in black-box studies is non-ignorable. Using data from a recent latent print study, we show that ignoring this missingness likely results in systematic underestimates of error rates. Finally, we offer concrete steps to overcome these limitations.