A Variance Decomposition Approach to Inconclusives in Forensic Black Box Studies

TL;DR

This paper introduces a variance decomposition method to analyze inconclusive results in forensic black box studies, providing insights into examiner variability and refining error rate estimates.

Contribution

It proposes a novel variance decomposition approach to better understand inconclusives and distinguish between examiner variability and other factors in forensic studies.

Findings

Variance decomposition reveals examiner variability contributions.

Error rates are significantly affected by how inconclusives are treated.

Study highlights the importance of nuanced analysis of inconclusives in forensic error estimation.

Abstract

In the US, `black box' studies are increasingly being used to estimate the error rate of forensic disciplines. A sample of forensic examiner participants are asked to evaluate a set of items whose source is known to the researchers but not to the participants. Participants are asked to make a source determination (typically an identification, exclusion, or some kind of inconclusive). We study inconclusives in two black box studies, one on fingerprints and one on bullets. Rather than treating all inconclusive responses as functionally correct (as is the practice in reported error rates in the two studies we address), irrelevant to reported error rates (as some would do), or treating them all as potential errors (as others would do), we propose that the overall pattern of inconclusives in a particular black box study can shed light on the proportion of inconclusives that are due to examiner variability. Raw item and examiner variances are computed, and compared with the results of a logistic regression model that takes account of which items were addressed by which examiner. The error rates reported in black box studies are substantially smaller than ``failure rate" analyses that take inconclusives into account. The magnitude of this difference is highly dependent on the particular study at hand.