Usefulness of data flow diagrams and large language models for security threat validation: a registered report

TL;DR

This study investigates how data flow diagrams and large language models can aid security threat validation, aiming to improve scalability and effectiveness in cybersecurity assessments through controlled experiments with practitioners.

Contribution

It introduces a controlled experiment framework to evaluate the impact of LLM-generated advice and data flow diagrams on threat validation effectiveness.

Findings

Pilot study with 41 MSc students informed experiment design

Preliminary evidence suggests LLM advice can assist threat validation

Study aims to extend findings with practitioner data

Abstract

The arrival of recent cybersecurity standards has raised the bar for security assessments in organizations, but existing techniques don't always scale well. Threat analysis and risk assessment are used to identify security threats for new or refactored systems. Still, there is a lack of definition-of-done, so identified threats have to be validated which slows down the analysis. Existing literature has focused on the overall performance of threat analysis, but no previous work has investigated how deep must the analysts dig into the material before they can effectively validate the identified security threats. We propose a controlled experiment with practitioners to investigate whether some analysis material (like LLM-generated advice) is better than none and whether more material (the system's data flow diagram and LLM-generated advice) is better than some material. In addition, we present key findings from running a pilot with 41 MSc students, which are used to improve the study design. Finally, we also provide an initial replication package, including experimental material and data analysis scripts and a plan to extend it to include new materials based on the final data collection campaign with practitioners (e.g., pre-screening questions).