Deep hierarchical reinforcement agents for automated penetration testing

TL;DR

This paper introduces HA-DRL, a hierarchical deep reinforcement learning architecture that improves the efficiency and stability of autonomous penetration testing by effectively managing large action spaces.

Contribution

The paper presents a novel hierarchical deep reinforcement learning architecture with algebraic action decomposition for automated penetration testing.

Findings

HA-DRL finds optimal attack policies faster.

HA-DRL demonstrates more stable learning compared to conventional deep Q-learning.

Effective handling of large discrete action spaces in cybersecurity simulations.

Abstract

Penetration testing the organised attack of a computer system in order to test existing defences has been used extensively to evaluate network security. This is a time consuming process and requires in-depth knowledge for the establishment of a strategy that resembles a real cyber-attack. This paper presents a novel deep reinforcement learning architecture with hierarchically structured agents called HA-DRL, which employs an algebraic action decomposition strategy to address the large discrete action space of an autonomous penetration testing simulator where the number of actions is exponentially increased with the complexity of the designed cybersecurity network. The proposed architecture is shown to find the optimal attacking policy faster and more stably than a conventional deep Q-learning agent which is commonly used as a method to apply artificial intelligence in automatic penetration testing.