Measuring Accuracy of Automated Parsing and Categorization Tools and Processes in Digital Investigations

TL;DR

This paper introduces a novel method for measuring the accuracy of digital forensic tools and processes by applying information retrieval metrics against a gold standard derived from expert analysis, enabling ongoing assessment of investigative reliability.

Contribution

It proposes a new accuracy measurement approach for forensic tools that accounts for errors in analysis processes using precision and recall metrics compared to a gold standard.

Findings

The method effectively evaluates tool accuracy against expert-derived standards.

Case studies demonstrate the ability to track accuracy changes over time.

The approach helps distinguish between tool and investigator errors.

Abstract

This work presents a method for the measurement of the accuracy of evidential artifact extraction and categorization tasks in digital forensic investigations. Instead of focusing on the measurement of accuracy and errors in the functions of digital forensic tools, this work proposes the application of information retrieval measurement techniques that allow the incorporation of errors introduced by tools and analysis processes. This method uses a `gold standard' that is the collection of evidential objects determined by a digital investigator from suspect data with an unknown ground truth. This work proposes that the accuracy of tools and investigation processes can be evaluated compared to the derived gold standard using common precision and recall values. Two example case studies are presented showing the measurement of the accuracy of automated analysis tools as compared to an in-depth analysis by an expert. It is shown that such measurement can allow investigators to determine changes in accuracy of their processes over time, and determine if such a change is caused by their tools or knowledge.