Subtleties in Bayesian decision-theoretic analysis for forensic findings: Notes on recent discussion of the role of validation study data in rational decision making

TL;DR

This paper discusses how using validation study data in forensic decisions can lose important information when simplified into error rates.

Contribution

The paper clarifies how different levels of detail in Bayesian decision theory affect the use of validation data in forensic science.

Findings

Aggregating validation study data into error rates can lead to a loss of information.

The usefulness of error rates in legal decisions depends on how propositions and findings are defined.

Discussions about validation data involve multiple layers of assumptions and abstractions.

Abstract

This technical note extends a recent discussion in this journal of the role of validation study data in rational decision making. One argument that has been made in this context, using elements of Bayesian decision theory, is that further aggregation of validation study data into error rates involves a loss of information that compromises rational inference and decision making and should therefore be discouraged. This technical note seeks to explain that this argument can be developed at different levels of detail, depending on the definition of the propositions of interest, the forensic findings to be evaluated (and hence the form of the likelihood ratio), and the characterization of the relative desirability of decision consequences. The analyses proposed here reveal the cascade of abstractions and assumptions into which discussions about the use of validation study results in forensic science have fallen. This reinforces the conclusion that further aggregation of validation study data into error rates is problematic. It also suggests that even if a definition of error rate(s) could be agreed upon and defensively quantified in a given application, we should rethink and possibly adjust our expectations about what exactly error rates can practically contribute to rational modes of reasoning and decision making in legal contexts.

Figures