Towards Leveraging LLMs to Generate Abstract Penetration Test Cases from Software Architecture

TL;DR

This paper proposes a novel approach to generate Abstract Penetration Test Cases from software architecture models using large language models, aiming to improve early security assessment and testing guidance.

Contribution

It introduces a metamodel for APTCs and demonstrates how LLMs can generate meaningful security test cases from architecture models.

Findings

Achieved up to 93% usefulness in generated APTCs

Achieved up to 86% correctness in generated APTCs

Supports early architecture-level security assessment

Abstract

Software architecture models capture early design decisions that strongly influence system quality attributes, including security. However, architecture-level security assessment and feedback are often absent in practice, allowing security weaknesses to propagate into later phases of the software development lifecycle and, in some cases, to remain undiscovered, ultimately leading to vulnerable systems. In this paper, we bridge this gap by proposing the generation of Abstract Penetration Test Cases (APTCs) from software architecture models as an input to support architecture-level security assessment. We first introduce a metamodel that defines the APTC concept, and then investigate the use of large language models with different prompting strategies to generate meaningful APTCs from architecture models. To design the APTC metamodel, we analyze relevant standards and state of the art using two criteria: (i) derivability from software architecture, and (ii) usability for both architecture security assessment and subsequent penetration testing. Building on this metamodel, we then proceed to generate APTCs from software architecture models. Our evaluation shows promising results, achieving up to 93\% usefulness and 86\% correctness, indicating that the generated APTCs can substantially support both architects (by highlighting security-critical design decisions) and penetration testers (by providing actionable testing guidance).