CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructions

TL;DR

This paper presents CyberPal.AI, a family of security-specialized LLMs trained with expert-curated cyber-security instructions and evaluated on a comprehensive benchmark, achieving up to 24% performance improvements.

Contribution

Introduction of SecKnowledge dataset and CyberPal.AI models, leveraging expert knowledge to enhance LLMs for cyber-security tasks.

Findings

Up to 24% performance improvement over baseline models

Development of SecKnowledge-Eval benchmark for cyber-security tasks

Demonstrated effectiveness of expert-driven instruction datasets

Abstract

Large Language Models (LLMs) have significantly advanced natural language processing (NLP), providing versatile capabilities across various applications. However, their application to complex, domain-specific tasks, such as cyber-security, often faces substantial challenges. In this study, we introduce SecKnowledge and CyberPal.AI to address these challenges and train security-expert LLMs. SecKnowledge is a domain-knowledge-driven cyber-security instruction dataset, meticulously designed using years of accumulated expert knowledge in the domain through a multi-phase generation process. CyberPal.AI refers to a family of LLMs fine-tuned using SecKnowledge, aimed at building security-specialized LLMs capable of answering and following complex security-related instructions. Additionally, we introduce SecKnowledge-Eval, a comprehensive and diverse cyber-security evaluation benchmark, composed of an extensive set of cyber-security tasks we specifically developed to assess LLMs in the field of cyber-security, along with other publicly available security benchmarks. Our results show a significant average improvement of up to 24% over the baseline models, underscoring the benefits of our expert-driven instruction dataset generation process. These findings contribute to the advancement of AI-based cyber-security applications, paving the way for security-expert LLMs that can enhance threat-hunting and investigation processes.