An Automated Framework for Cybersecurity Policy Compliance Assessment Against Security Control Standards

TL;DR

PROPARAG is an autonomous framework that evaluates organizational cybersecurity policies against security controls using large language models, providing gap analysis and grounded recommendations.

Contribution

It introduces PROPARAG, a novel LLM-based system for automated, detailed cybersecurity policy compliance assessment against standard controls.

Findings

Achieves high F1 scores of 88.54 and 82.31 on two organizational corpora.

Effectively identifies policy gaps and generates grounded recommendations.

Demonstrates the feasibility of LLM-powered autonomous policy assessment.

Abstract

Organizational cybersecurity policies are often examined to determine whether they adequately comply standard security controls. This task is difficult because control statements are abstract, whereas policy documents describe governance practices in varied natural language. As a result, policy-based control assessment is time-consuming, difficult to standardize, and often difficult to document in a traceable manner. To address this gap, we present PROPARAG, an audit support approach for evaluating organizational cybersecurity policies against security controls autonomously. For each control, the approach retrieves relevant policy evidence, assesses coverage, identifies missing elements, and generates supporting explanations and recommendations. We evaluate PROPARAG on two real-world organizational policy corpora using 1,007 NIST SP 800-53 controls across both closed-source and open-source large language models (LLMs). The framework achieves F1 scores of 88.54 on OrgA and 82.31 on OrgB. The evaluation also shows that PROPARAG identifies relevant gaps in documented organizational policies and generates grounded recommendations for each identified gap. This research provides foundation for LLM-powered autonomous control-level assessment of organizational cybersecurity policies.