Automated Adversary Emulation for Cyber-Physical Systems via Reinforcement Learning

TL;DR

This paper introduces an automated, reinforcement learning-based method for adversary emulation in cyber-physical systems, enabling efficient and optimal attack sequence determination amidst complex dynamics and uncertainties.

Contribution

It formulates a hybrid attack graph as an MDP and applies RL techniques to automate and optimize adversary emulation in CPS, improving over manual approaches.

Findings

RL methods outperform greedy attack algorithms in solution quality

The approach effectively models both cyber and physical attack components

Numerical study demonstrates practical applicability in building sensor deception attacks

Abstract

Adversary emulation is an offensive exercise that provides a comprehensive assessment of a system's resilience against cyber attacks. However, adversary emulation is typically a manual process, making it costly and hard to deploy in cyber-physical systems (CPS) with complex dynamics, vulnerabilities, and operational uncertainties. In this paper, we develop an automated, domain-aware approach to adversary emulation for CPS. We formulate a Markov Decision Process (MDP) model to determine an optimal attack sequence over a hybrid attack graph with cyber (discrete) and physical (continuous) components and related physical dynamics. We apply model-based and model-free reinforcement learning (RL) methods to solve the discrete-continuous MDP in a tractable fashion. As a baseline, we also develop a greedy attack algorithm and compare it with the RL procedures. We summarize our findings through a numerical study on sensor deception attacks in buildings to compare the performance and solution quality of the proposed algorithms.