A Variance Decomposition Approach to Inconclusives in Forensic Black Box Studies
Amanda Luby, Joseph B. Kadane

TL;DR
This paper introduces a variance decomposition method to analyze inconclusive results in forensic black box studies, providing insights into examiner variability and refining error rate estimates.
Contribution
It proposes a novel variance decomposition approach to better understand inconclusives and distinguish between examiner variability and other factors in forensic studies.
Findings
Variance decomposition reveals examiner variability contributions.
Error rates are significantly affected by how inconclusives are treated.
Study highlights the importance of nuanced analysis of inconclusives in forensic error estimation.
Abstract
In the US, `black box' studies are increasingly being used to estimate the error rate of forensic disciplines. A sample of forensic examiner participants are asked to evaluate a set of items whose source is known to the researchers but not to the participants. Participants are asked to make a source determination (typically an identification, exclusion, or some kind of inconclusive). We study inconclusives in two black box studies, one on fingerprints and one on bullets. Rather than treating all inconclusive responses as functionally correct (as is the practice in reported error rates in the two studies we address), irrelevant to reported error rates (as some would do), or treating them all as potential errors (as others would do), we propose that the overall pattern of inconclusives in a particular black box study can shed light on the proportion of inconclusives that are due to…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
