Hey GPT-OSS, Looks Like You Got It -- Now Walk Me Through It! An Assessment of the Reasoning Language Models Chain of Thought Mechanism for Digital Forensics

TL;DR

This paper evaluates the potential of reasoning language models, specifically gpt-oss, to improve explainability and validation in digital forensics tasks, revealing moderate success at medium reasoning levels.

Contribution

It is the first study to assess reasoning language models like gpt-oss for digital forensics, focusing on their explainability and validation capabilities.

Findings

Reasoning component helps explain and validate outputs at medium reasoning levels.

Higher reasoning levels do not significantly improve response quality.

A new quantitative metric was developed for evaluation.

Abstract

The use of large language models in digital forensics has been widely explored. Beyond identifying potential applications, research has also focused on optimizing model performance for forensic tasks through fine-tuning. However, limited result explainability reduces their operational and legal usability. Recently, a new class of reasoning language models has emerged, designed to handle logic-based tasks through an `internal reasoning' mechanism. Yet, users typically see only the final answer, not the underlying reasoning. One of these reasoning models is gpt-oss, which can be deployed locally, providing full access to its underlying reasoning process. This article presents the first investigation into the potential of reasoning language models for digital forensics. Four test use cases are examined to assess the usability of the reasoning component in supporting result explainability. The evaluation combines a new quantitative metric with qualitative analysis. Findings show that the reasoning component aids in explaining and validating language model outputs in digital forensics at medium reasoning levels, but this support is often limited, and higher reasoning levels do not enhance response quality.