Psychometric Analysis of Forensic Examiner Behavior

TL;DR

This paper introduces a psychometric modeling approach using IRT and Bayesian models to analyze and account for variability in forensic examiner decision-making and task difficulty, aiming to improve understanding of source identification uncertainty.

Contribution

It applies IRT and Bayesian psychometric models to forensic examiner behavior, addressing variability among examiners and task difficulty, which was not previously modeled in this context.

Findings

IRT models reveal examiner performance variability

Bayesian models incorporate task difficulty into analysis

Enhanced understanding of decision-making uncertainty

Abstract

Forensic science often involves the comparison of crime-scene evidence to a known-source sample to determine if the evidence and the reference sample came from the same source. Even as forensic analysis tools become increasingly objective and automated, final source identifications are often left to individual examiners' interpretation of the evidence. Each source identification relies on judgements about the features and quality of the crime-scene evidence that may vary from one examiner to the next. The current approach to characterizing uncertainty in examiners' decision-making has largely centered around the calculation of error rates aggregated across examiners and identification tasks, without taking into account these variations in behavior. We propose a new approach using IRT and IRT-like models to account for differences among examiners and additionally account for the varying difficulty among source identification tasks. In particular, we survey some recent advances (Luby, 2019a) in the application of Bayesian psychometric models, including simple Rasch models as well as more elaborate decision tree models, to fingerprint examiner behavior.