Accuracy, Repeatability, and Reproducibility of Firearm Comparisons Part 1: Accuracy

TL;DR

This study evaluates the accuracy, repeatability, and reproducibility of firearm comparisons by forensic examiners, revealing low error rates and highlighting variability among examiners across different firearm types.

Contribution

It provides a comprehensive, robust assessment of examiner performance in firearm identification, including detailed error rate estimates and confidence intervals, using a large dataset and advanced statistical modeling.

Findings

False positive error rates are below 1% for bullets and cartridge cases.

False negative rates are approximately 2-3% for bullets and cartridge cases.

Error probabilities vary significantly among examiners.

Abstract

Researchers at the Ames Laboratory-USDOE and the Federal Bureau of Investigation (FBI) conducted a study to assess the performance of forensic examiners in firearm investigations. The study involved three different types of firearms and 173 volunteers who compared both bullets and cartridge cases. The total number of comparisons reported is 20,130, allocated to assess accuracy (8,640), repeatability (5,700), and reproducibility (5,790) of the evaluations made by participating examiners. The overall false positive error rate was estimated as 0.656% and 0.933% for bullets and cartridge cases, respectively, while the rate of false negatives was estimated as 2.87% and 1.87% for bullets and cartridge cases, respectively. Because chi-square tests of independence strongly suggest that error probabilities are not the same for each examiner, these are maximum likelihood estimates based on the beta-binomial probability model and do not depend on an assumption of equal examiner-specific error rates. Corresponding 95% confidence intervals are (0.305%,1.42%) and (0.548%,1.57%) for false positives for bullets and cartridge cases, respectively, and (1.89%,4.26%) and (1.16%,2.99%) for false negatives for bullets and cartridge cases, respectively. These results are based on data representing all controlled conditions considered, including different firearm manufacturers, sequence of manufacture, and firing separation between unknown and known comparison specimens. The results are consistent with those of prior studies, despite its more robust design and challenging specimens.