Probabilistic genotyping code review and testing

TL;DR

This paper reviews the detection of miscodes in probabilistic genotyping software, emphasizing the importance of testing and use in identifying errors, and discusses how code review complements these methods.

Contribution

It highlights the limitations of code review alone in finding miscodes and advocates for combined testing, use, and review strategies in PG software validation.

Findings

Miscode detection often relies on testing and use, not just code review.

Code review aids in diagnosing issues once anomalies are detected.

Historical miscodes from other industries illustrate the importance of comprehensive validation.

Abstract

We discuss a range of miscodes found in probabilistic genotyping (PG) software and from other industries that have been reported in the literature and have been used to inform PG admissibility hearings. Every instance of the discovery of a miscode in PG software with which we have been associated has occurred either because of testing, use, or repeat calculation of results either by us or other users. In all cases found during testing or use something has drawn attention to an anomalous result. Intelligent investigation has led to the examination of a small section of the code and detection of the miscode. Previously, three instances from other industries quoted by the Electronic Frontier Foundation Amicus brief as part of a PG admissibility hearing (atmospheric ozone, NIMIS, and VW) and two previous examples raised in relation to PG admissibility (Kerberos and Therac-25) were presented as examples of miscodes and how an extensive code review could have resolved these situations. However, we discuss how these miscodes might not have been discovered through code review alone. These miscodes could only have been detected through use of the software or through testing. Once the symptoms of the miscode(s) have been detected, a code review serves as a beneficial approach to try and diagnose to the issue.