On the Ethics of Using LLMs for Offensive Security

TL;DR

This paper reviews how offensive security research using LLMs addresses ethical concerns, highlighting prevalent practices, gaps, and the balance between innovation and responsibility.

Contribution

It provides an analysis of ethical discourse in LLM-based offensive security research, identifying trends, best practices, and areas needing improvement.

Findings

86.6% of reviewed prototypes mention ethical considerations

Main motivation is to enable broader penetration-testing access

Research aims to prepare defenders for AI-guided attacks

Abstract

Large Language Models (LLMs) have rapidly evolved over the past few years and are currently evaluated for their efficacy within the domain of offensive cyber-security. While initial forays showcase the potential of LLMs to enhance security research, they also raise critical ethical concerns regarding the dual-use of offensive security tooling. This paper analyzes a set of papers that leverage LLMs for offensive security, focusing on how ethical considerations are expressed and justified in their work. The goal is to assess the culture of AI in offensive security research regarding ethics communication, highlighting trends, best practices, and gaps in current discourse. We provide insights into how the academic community navigates the fine line between innovation and ethical responsibility. Particularly, our results show that 13 of 15 reviewed prototypes (86.6\%) mentioned ethical considerations and are thus aware of the potential dual-use of their research. Main motivation given for the research was allowing broader access to penetration-testing as well as preparing defenders for AI-guided attackers.