Towards Production-Worthy Simulation for Autonomous Cyber Operations

TL;DR

This paper enhances a cybersecurity simulation environment to better train reinforcement learning agents by adding realistic actions and modifying training signals, demonstrating improved agent training capabilities.

Contribution

The paper introduces new actions and reward modifications to an existing cybersecurity simulation environment, improving its realism and training effectiveness for RL agents.

Findings

Extended CybORG with Patch, Isolate, Unisolate actions

Modified reward signals and feature space

Trained DQN and PPO agents successfully

Abstract

Simulated environments have proven invaluable in Autonomous Cyber Operations (ACO) where Reinforcement Learning (RL) agents can be trained without the computational overhead of emulation. These environments must accurately represent cybersecurity scenarios while producing the necessary signals to support RL training. In this study, we present a framework where we first extend CybORG's Cage Challenge 2 environment by implementing three new actions: Patch, Isolate, and Unisolate, to better represent the capabilities available to human operators in real-world settings. We then propose a design for agent development where we modify the reward signals and the agent's feature space to enhance training performance. To validate these modifications, we train DQN and PPO agents in the updated environment. Our study demonstrates that CybORG can be extended with additional realistic functionality, while maintaining its ability to generate informative training signals for RL agents.