SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecurity

TL;DR

This paper systematically identifies common methodological pitfalls in applying deep reinforcement learning to cybersecurity, analyzing 66 papers and demonstrating their practical impact through controlled experiments, with recommendations for more robust systems.

Contribution

It is the first comprehensive systematic review highlighting 11 pitfalls in DRL for cybersecurity and providing actionable guidance to improve research rigor and deployment.

Findings

Over five pitfalls per paper on average

Demonstrated impact of pitfalls in cyber defense and malware creation

Provided actionable recommendations for each identified pitfall

Abstract

Deep Reinforcement Learning (DRL) has achieved remarkable success in domains requiring sequential decision-making, motivating its application to cybersecurity problems. However, transitioning DRL from laboratory simulations to bespoke cyber environments can introduce numerous issues. This is further exacerbated by the often adversarial, non-stationary, and partially-observable nature of most cybersecurity tasks. In this paper, we identify and systematize 11 methodological pitfalls that frequently occur in DRL for cybersecurity (DRL4Sec) literature across the stages of environment modeling, agent training, performance evaluation, and system deployment. By analyzing 66 significant DRL4Sec papers (2018-2025), we quantify the prevalence of each pitfall and find an average of over five pitfalls per paper. We demonstrate the practical impact of these pitfalls using controlled experiments in (i) autonomous cyber defense, (ii) adversarial malware creation, and (iii) web security testing environments. Finally, we provide actionable recommendations for each pitfall to support the development of more rigorous and deployable DRL-based security systems.