Methodological Problems in Every Black-Box Study of Forensic Firearm Comparisons

TL;DR

This paper critically reviews forensic firearm comparison studies, revealing fundamental methodological flaws that undermine their scientific validity and impede accurate assessment of examiner performance.

Contribution

It identifies major design and analysis flaws in existing studies and offers recommendations to improve future research in forensic firearm comparisons.

Findings

All reviewed studies have serious methodological flaws.

Current studies do not reliably estimate examiner error rates.

Statements on firearm origin lack scientific support due to study flaws.

Abstract

Reviews conducted by the National Academy of Sciences (2009) and the President's Council of Advisors on Science and Technology (2016) concluded that the field of forensic firearm comparisons has not been demonstrated to be scientifically valid. Scientific validity requires adequately designed studies of firearm examiner performance in terms of accuracy, repeatability, and reproducibility. Researchers have performed ``black-box'' studies with the goal of estimating these performance measures. As statisticians with expertise in experimental design, we conducted a literature search of such studies to date and then evaluated the design and statistical analysis methods used in each study. Our conclusion is that all studies in our literature search have methodological flaws that are so grave that they render the studies invalid, that is, incapable of establishing scientific validity of the field of firearms examination. Notably, error rates among firearms examiners, both collectively and individually, remain unknown. Therefore, statements about the common origin of bullets or cartridge cases that are based on examination of ``individual" characteristics do not have a scientific basis. We provide some recommendations for the design and analysis of future studies.