Assessing Spear-Phishing Website Generation in Large Language Model Coding Agents

TL;DR

This paper evaluates large language models' ability to generate spear-phishing websites, highlighting risks and providing a dataset to aid in understanding and defending against such misuse in cybersecurity.

Contribution

It introduces a dataset of 200 spear-phishing website code bases and analyzes LLMs' capabilities and tendencies to produce malicious code, a novel focus in cybersecurity assessment of LLMs.

Findings

Certain LLM metrics correlate with phishing site generation ability

Models vary significantly in willingness to produce malicious code

Dataset enables further research on LLM misuse in cyberattacks

Abstract

Large Language Models are expanding beyond being a tool humans use and into independent agents that can observe an environment, reason about solutions to problems, make changes that impact those environments, and understand how their actions impacted their environment. One of the most common applications of these LLM Agents is in computer programming, where agents can successfully work alongside humans to generate code while controlling programming environments or networking systems. However, with the increasing ability and complexity of these agents comes dangers about the potential for their misuse. A concerning application of LLM agents is in the domain cybersecurity, where they have the potential to greatly expand the threat imposed by attacks such as social engineering. This is due to the fact that LLM Agents can work autonomously and perform many tasks that would normally require time and effort from skilled human programmers. While this threat is concerning, little attention has been given to assessments of the capabilities of LLM coding agents in generating code for social engineering attacks. In this work we compare different LLMs in their ability and willingness to produce potentially dangerous code bases that could be misused by cyberattackers. The result is a dataset of 200 website code bases and logs from 40 different LLM coding agents. Analysis of models shows which metrics of LLMs are more and less correlated with performance in generating spear-phishing sites. Our analysis and the dataset we present will be of interest to researchers and practitioners concerned in defending against the potential misuse of LLMs in spear-phishing.