Good News for Script Kiddies? Evaluating Large Language Models for Automated Exploit Generation

TL;DR

This study systematically evaluates large language models' ability to generate exploits, revealing high cooperativeness in some models but limited success in generating exploits for refactored, more secure labs.

Contribution

Introduces a benchmark with refactored security labs and an LLM-based attacker to assess LLMs' effectiveness in automated exploit generation.

Findings

GPT-4 and GPT-4o show high cooperativeness

Llama3 is most resistant to exploitation

No model successfully exploits refactored labs

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in code-related tasks, raising concerns about their potential for automated exploit generation (AEG). This paper presents the first systematic study on LLMs' effectiveness in AEG, evaluating both their cooperativeness and technical proficiency. To mitigate dataset bias, we introduce a benchmark with refactored versions of five software security labs. Additionally, we design an LLM-based attacker to systematically prompt LLMs for exploit generation. Our experiments reveal that GPT-4 and GPT-4o exhibit high cooperativeness, comparable to uncensored models, while Llama3 is the most resistant. However, no model successfully generates exploits for refactored labs, though GPT-4o's minimal errors highlight the potential for LLM-driven AEG advancements.