Automating Security Audit Using Large Language Model based Agent: An Exploration Experiment

TL;DR

This paper explores using GPT-4 with Langchain as an autonomous agent to automate password policy compliance audits on Windows, showing improved efficiency over manual methods and potential for broader security applications.

Contribution

It introduces a novel framework leveraging LLMs for automated security audits, specifically demonstrating its application in password policy compliance checks.

Findings

GPT-4 accurately flagged password violations

The agent was more efficient than manual audits

Framework shows potential for real-time security monitoring

Abstract

In the current rapidly changing digital environment, businesses are under constant stress to ensure that their systems are secured. Security audits help to maintain a strong security posture by ensuring that policies are in place, controls are implemented, gaps are identified for cybersecurity risks mitigation. However, audits are usually manual, requiring much time and costs. This paper looks at the possibility of developing a framework to leverage Large Language Models (LLMs) as an autonomous agent to execute part of the security audit, namely with the field audit. password policy compliance for Windows operating system. Through the conduct of an exploration experiment of using GPT-4 with Langchain, the agent executed the audit tasks by accurately flagging password policy violations and appeared to be more efficient than traditional manual audits. Despite its potential limitations in operational consistency in complex and dynamic environment, the framework suggests possibilities to extend further to real-time threat monitoring and compliance checks.