From nuclear safety to LLM security: Applying non-probabilistic risk management strategies to build safe and secure LLM-powered systems

TL;DR

This paper explores the application of non-probabilistic risk management strategies, originally used in fields like nuclear engineering, to enhance the safety and security of large language models against complex and adaptive threats.

Contribution

It introduces over 100 non-probabilistic risk management strategies, categorized and adapted for LLM security, providing a practical workflow for implementation.

Findings

Mapped strategies to LLM security and AI safety

Developed an LLM-powered risk management workflow

Demonstrated potential to improve safety and security

Abstract

Large language models (LLMs) offer unprecedented and growing capabilities, but also introduce complex safety and security challenges that resist conventional risk management. While conventional probabilistic risk analysis (PRA) requires exhaustive risk enumeration and quantification, the novelty and complexity of these systems make PRA impractical, particularly against adaptive adversaries. Previous research found that risk management in various fields of engineering such as nuclear or civil engineering is often solved by generic (i.e. field-agnostic) strategies such as event tree analysis or robust designs. Here we show how emerging risks in LLM-powered systems could be met with 100+ of these non-probabilistic strategies to risk management, including risks from adaptive adversaries. The strategies are divided into five categories and are mapped to LLM security (and AI safety more broadly). We also present an LLM-powered workflow for applying these strategies and other workflows suitable for solution architects. Overall, these strategies could contribute (despite some limitations) to security, safety and other dimensions of responsible AI.