Reinforcement Learning for Automated Cybersecurity Penetration Testing

TL;DR

This paper introduces a reinforcement learning approach combined with geometric deep learning to automate and optimize cybersecurity penetration testing on web applications, improving vulnerability detection efficiency.

Contribution

It presents a novel reinforcement learning framework that leverages geometric deep learning to enhance automated web security testing, reducing manual effort and increasing effectiveness.

Findings

Maximized vulnerability detection rate

Reduced number of testing steps

Validated on real-world vulnerable web pages

Abstract

This paper aims to provide an innovative machine learning-based solution to automate security testing tasks for web applications, ensuring the correct functioning of all components while reducing project maintenance costs. Reinforcement Learning is proposed to select and prioritize tools and optimize the testing path. The presented approach utilizes a simulated webpage along with its network topology to train the agent. Additionally, the model leverages Geometric Deep Learning to create priors that reduce the search space and improve learning convergence. The validation and testing process was conducted on real-world vulnerable web pages commonly used by human hackers for learning. As a result of this study, a reinforcement learning algorithm was developed that maximizes the number of vulnerabilities found while minimizing the number of steps required