SOCpilot: Verifying Policy Compliance for LLM-Assisted Incident Response

TL;DR

SOCpilot is a framework that verifies LLM-generated incident response plans against compliance policies, ensuring mandatory steps and approval gates are respected, with evaluation on real SOC incidents.

Contribution

It introduces a measurable compliance verification method for LLM-assisted incident response plans, including a fixed artifact and an open-source verifier tool.

Findings

466 non-compliant actions removed by verifier

Plans remain consistent across reruns

Artifact enables zero-cost compliance checks

Abstract

Security operations centers (SOCs) are beginning to use large language models (LLMs) as copilots to draft incident-response plans. These plans may include actions that are valid per the catalog but still violate mandatory steps, required ordering, or approval gates before analyst review. SOCpilot makes this compliance question measurable at the plan boundary. It fixes the incident package, action catalog, policy rules, verifier, and public evidence surface. Next, it verifies the copilot's proposed action trace. We evaluate two LLM providers on 200 real incidents from an anonymized production SOC in a financial-sector case study. We compare their plans to paired analyst-authored references from the same security orchestration, automation, and response (SOAR) cases. An identical inline policy text moves the two providers in opposite directions. A deterministic verifier removes 466 non-compliant, approval-gated actions, without reducing baseline-task recall. Aggregate rates remain stable across 3 reruns of the fixed corpus. The official evidence focuses on approval-gated decisions regarding recovery and containment. Separately, the artifact exposes zero-cost readiness checks for mandatory and ordering repairs. We release the runnable artifact so independent reviewers can rederive the public results without access to private incident data.