Goal-Driven Risk Assessment for LLM-Powered Systems: A Healthcare Case Study

TL;DR

This paper introduces a goal-driven risk assessment method using attack trees for LLM-powered healthcare systems, addressing security challenges and enhancing secure-by-design practices.

Contribution

It presents a structured approach to threat modeling with detailed attack vectors, attack paths, and contextualization for LLM-based systems, demonstrated through a healthcare case study.

Findings

Harmonizes state-of-the-art LLM attacks with traditional threats

Identifies detailed attack paths and vectors for healthcare LLM systems

Enhances risk prioritization through structured threat assessment

Abstract

While incorporating LLMs into systems offers significant benefits in critical application areas such as healthcare, new security challenges emerge due to the potential cyber kill chain cycles that combine adversarial model, prompt injection and conventional cyber attacks. Threat modeling methods enable the system designers to identify potential cyber threats and the relevant mitigations during the early stages of development. Although the cyber security community has extensive experience in applying these methods to software-based systems, the elicited threats are usually abstract and vague, limiting their effectiveness for conducting proper likelihood and impact assessments for risk prioritization, especially in complex systems with novel attacks surfaces, such as those involving LLMs. In this study, we propose a structured, goal driven risk assessment approach that contextualizes the threats with detailed attack vectors, preconditions, and attack paths through the use of attack trees. We demonstrate the proposed approach on a case study with an LLM agent-based healthcare system. This study harmonizes the state-of-the-art attacks to LLMs with conventional ones and presents possible attack paths applicable to similar systems. By providing a structured risk assessment, this study makes a significant contribution to the literature and advances the secure-by-design practices in LLM-based systems.