Depending on yourself when you should: Mentoring LLM with RL agents to become the master in cybersecurity games

TL;DR

This paper presents SecurityBot, an innovative LLM mentored by pre-trained RL agents, to enhance cybersecurity game performance through modules for behavior, memory, reflection, and collaboration, demonstrating significant improvements over standalone models.

Contribution

Introduces SecurityBot, a novel framework combining LLM and RL agents with modules for behavior, memory, reflection, and collaboration to improve cybersecurity game performance.

Findings

SecurityBot outperforms standalone LLM and RL agents.

Modules enable effective behavior generation and experience accumulation.

Collaboration mechanism enhances decision-making in cybersecurity scenarios.

Abstract

Integrating LLM and reinforcement learning (RL) agent effectively to achieve complementary performance is critical in high stake tasks like cybersecurity operations. In this study, we introduce SecurityBot, a LLM agent mentored by pre-trained RL agents, to support cybersecurity operations. In particularly, the LLM agent is supported with a profile module to generated behavior guidelines, a memory module to accumulate local experiences, a reflection module to re-evaluate choices, and an action module to reduce action space. Additionally, it adopts the collaboration mechanism to take suggestions from pre-trained RL agents, including a cursor for dynamic suggestion taken, an aggregator for multiple mentors' suggestions ranking and a caller for proactive suggestion asking. Building on the CybORG experiment framework, our experiences show that SecurityBot demonstrates significant performance improvement compared with LLM or RL standalone, achieving the complementary performance in the cybersecurity games.