Interpreting Agent Behaviors in Reinforcement-Learning-Based Cyber-Battle Simulation Platforms

TL;DR

This paper analyzes reinforcement learning agents in a cyber defense simulation, providing interpretability of their behaviors, evaluating effectiveness of actions and decoys, and discussing the realism of the simulation platform.

Contribution

It introduces methods to interpret complex agent behaviors in cyber simulations and evaluates the impact of decoys and action effectiveness within the CAGE Challenge environment.

Findings

Defenders clear infiltrations within 1-2 steps in most cases.

Certain actions are between 40% and 99% ineffective.

Decoys block up to 94% of exploits that would grant privileged access.

Abstract

We analyze two open source deep reinforcement learning agents submitted to the CAGE Challenge 2 cyber defense challenge, where each competitor submitted an agent to defend a simulated network against each of several provided rules-based attack agents. We demonstrate that one can gain interpretability of agent successes and failures by simplifying the complex state and action spaces and by tracking important events, shedding light on the fine-grained behavior of both the defense and attack agents in each experimental scenario. By analyzing important events within an evaluation episode, we identify patterns in infiltration and clearing events that tell us how well the attacker and defender played their respective roles; for example, defenders were generally able to clear infiltrations within one or two timesteps of a host being exploited. By examining transitions in the environment's state caused by the various possible actions, we determine which actions tended to be effective and which did not, showing that certain important actions are between 40% and 99% ineffective. We examine how decoy services affect exploit success, concluding for instance that decoys block up to 94% of exploits that would directly grant privileged access to a host. Finally, we discuss the realism of the challenge and ways that the CAGE Challenge 4 has addressed some of our concerns.