A Re-analysis of Repeatability and Reproducibility in the Ames-USDOE-FBI Study

TL;DR

This paper critically re-analyzes a key study on forensic firearms examination, revealing that its conclusions about repeatability and reproducibility are misleading due to misinterpretation of statistical measures.

Contribution

It clarifies the proper interpretation of agreement metrics in forensic studies, challenging previous claims of satisfactory reliability.

Findings

Observed agreement does not necessarily indicate high reliability.

Proper use of expected agreement shows lower repeatability and reproducibility.

The study questions the validity of prior conclusions on forensic firearm exam reliability.

Abstract

Forensic firearms identification, the determination by a trained firearms examiner as to whether or not bullets or cartridges came from a common weapon, has long been a mainstay in the criminal courts. Reliability of forensic firearms identification has been challenged in the general scientific community, and, in response, several studies have been carried out aimed at showing that firearms examination is accurate, that is, has low error rates. Less studied has been the question of consistency, of. whether two examinations of the same bullets or cartridge cases come to the same conclusion, carried out by an examiner on separate occasions -- intrarater reliability or repeatability -- or by two examiners -- interrater reliability or reproducibility. One important study, described in a 2020 Report by the Ames Laboratory-USDOE to the Federal Bureau of Investigation, went beyond considerations of accuracy to investigate firearms examination repeatability and reproducibility. The Report's conclusions were paradoxical. The observed agreement of examiners with themselves or with other examiners appears mediocre. However, the study concluded repeatability and reproducibility are satisfactory, on grounds that the observed agreement exceeds a quantity called the expected agreement. We find that appropriately employing expected agreement as it was intended does not suggest satisfactory repeatability and reproducibility, but the opposite.