On Automating Security Policies with Contemporary LLMs

TL;DR

This paper introduces a framework that uses large language models combined with retrieval-augmented generation to automate security policy enforcement, improving accuracy in attack mitigation tasks.

Contribution

It presents a novel system integrating LLMs with retrieval techniques to automate and enhance security policy compliance in complex environments.

Findings

RAG improves precision, recall, and F1-score over baseline methods.

The system effectively decomposes high-level policies into actionable API calls.

Empirical results show significant performance gains in security policy automation.

Abstract

The complexity of modern computing environments and the growing sophistication of cyber threats necessitate a more robust, adaptive, and automated approach to security enforcement. In this paper, we present a framework leveraging large language models (LLMs) for automating attack mitigation policy compliance through an innovative combination of in-context learning and retrieval-augmented generation (RAG). We begin by describing how our system collects and manages both tool and API specifications, storing them in a vector database to enable efficient retrieval of relevant information. We then detail the architectural pipeline that first decomposes high-level mitigation policies into discrete tasks and subsequently translates each task into a set of actionable API calls. Our empirical evaluation, conducted using publicly available CTI policies in STIXv2 format and Windows API documentation, demonstrates significant improvements in precision, recall, and F1-score when employing RAG compared to a non-RAG baseline.