On-Premise SLMs vs. Commercial LLMs: Prompt Engineering and Incident Classification in SOCs and CSIRTs

TL;DR

This paper compares open-source and proprietary LLMs for security incident classification, highlighting the trade-offs in accuracy, privacy, cost, and data sovereignty using prompt engineering techniques.

Contribution

It provides a comprehensive evaluation of open-source models against commercial LLMs in security incident classification with novel prompt-engineering methods.

Findings

Proprietary models achieve higher accuracy.

Open-source models offer better privacy and cost benefits.

Prompt engineering improves classification performance.

Abstract

In this study, we evaluate open-source models for security incident classification, comparing them with proprietary models. We utilize a dataset of anonymized real incidents, categorized according to the NIST SP 800-61r3 taxonomy and processed using five prompt-engineering techniques (PHP, SHP, HTP, PRP, and ZSL). The results indicate that, although proprietary models still exhibit higher accuracy, locally deployed open-source models provide advantages in privacy, cost-effectiveness, and data sovereignty.