How Good LLM-Generated Password Policies Are?

TL;DR

This paper evaluates the reliability of Large Language Models in generating password policies for cybersecurity, highlighting their current limitations in consistency and accuracy.

Contribution

It systematically assesses LLMs' ability to produce accurate, consistent password policy configurations from natural language prompts and documentation.

Findings

LLMs show significant inconsistency in password policy generation.

Using official documentation improves LLM accuracy.

Current LLMs face challenges in security-critical configuration tasks.

Abstract

Generative AI technologies, particularly Large Language Models (LLMs), are rapidly being adopted across industry, academia, and government sectors, owing to their remarkable capabilities in natural language processing. However, despite their strengths, the inconsistency and unpredictability of LLM outputs present substantial challenges, especially in security-critical domains such as access control. One critical issue that emerges prominently is the consistency of LLM-generated responses, which is paramount for ensuring secure and reliable operations. In this paper, we study the application of LLMs within the context of Cybersecurity Access Control Systems. Specifically, we investigate the consistency and accuracy of LLM-generated password policies, translating natural language prompts into executable pwquality$.$conf configuration files. Our experimental methodology adopts two distinct approaches: firstly, we utilize pre-trained LLMs to generate configuration files purely from natural language prompts without additional guidance. Secondly, we provide these models with official pwquality$.$conf documentation to serve as an informative baseline. We systematically assess the soundness, accuracy, and consistency of these AI-generated configurations. Our findings underscore significant challenges in the current generation of LLMs and contribute valuable insights into refining the deployment of LLMs in Access Control Systems.