An Empirical Game-Theoretic Analysis of Autonomous Cyber-Defence Agents

TL;DR

This paper employs an empirical game-theoretic approach to analyze deep reinforcement learning-based autonomous cyber-defense agents, introducing reward shaping and multiple response oracles to improve evaluation and robustness.

Contribution

It introduces a potential-based reward shaping method and extends the double oracle algorithm to multiple response oracles for comprehensive evaluation of ACD-DRL approaches.

Findings

Reward shaping accelerates the double oracle process.

Extended framework enables holistic evaluation of multiple ACD approaches.

Empirical analysis demonstrates improved robustness of ACD agents.

Abstract

The recent rise in increasingly sophisticated cyber-attacks raises the need for robust and resilient autonomous cyber-defence (ACD) agents. Given the variety of cyber-attack tactics, techniques and procedures (TTPs) employed, learning approaches that can return generalisable policies are desirable. Meanwhile, the assurance of ACD agents remains an open challenge. We address both challenges via an empirical game-theoretic analysis of deep reinforcement learning (DRL) approaches for ACD using the principled double oracle (DO) algorithm. This algorithm relies on adversaries iteratively learning (approximate) best responses against each others' policies; a computationally expensive endeavour for autonomous cyber operations agents. In this work we introduce and evaluate a theoretically-sound, potential-based reward shaping approach to expedite this process. In addition, given the increasing number of open-source ACD-DRL approaches, we extend the DO formulation to allow for multiple response oracles (MRO), providing a framework for a holistic evaluation of ACD approaches.