Act as a Honeytoken Generator! An Investigation into Honeytoken Generation with Large Language Models

TL;DR

This paper explores using large language models to automate the creation of diverse honeytokens for cybersecurity, demonstrating their effectiveness and variability across different models and prompt structures.

Contribution

It systematically investigates prompt-based honeytoken generation with LLMs, addressing limitations of existing automated methods and evaluating performance across multiple models.

Findings

GPT-3.5 generates honeywords less distinguishable from real passwords.

Optimal prompts vary between different LLMs.

Large language models can produce a wide variety of honeytokens.

Abstract

With the increasing prevalence of security incidents, the adoption of deception-based defense strategies has become pivotal in cyber security. This work addresses the challenge of scalability in designing honeytokens, a key component of such defense mechanisms. The manual creation of honeytokens is a tedious task. Although automated generators exists, they often lack versatility, being specialized for specific types of honeytokens, and heavily rely on suitable training datasets. To overcome these limitations, this work systematically investigates the approach of utilizing Large Language Models (LLMs) to create a variety of honeytokens. Out of the seven different honeytoken types created in this work, such as configuration files, databases, and log files, two were used to evaluate the optimal prompt. The generation of robots.txt files and honeywords was used to systematically test 210 different prompt structures, based on 16 prompt building blocks. Furthermore, all honeytokens were tested across different state-of-the-art LLMs to assess the varying performance of different models. Prompts performing optimally on one LLMs do not necessarily generalize well to another. Honeywords generated by GPT-3.5 were found to be less distinguishable from real passwords compared to previous methods of automated honeyword generation. Overall, the findings of this work demonstrate that generic LLMs are capable of creating a wide array of honeytokens using the presented prompt structures.