Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Training

TL;DR

This paper introduces Primus, a comprehensive set of open-source datasets for training cybersecurity LLMs, significantly improving their performance on benchmarks and security certifications.

Contribution

It provides the first extensive collection of cybersecurity datasets for all training stages, enabling better LLM pretraining and fine-tuning in cybersecurity.

Findings

15.9% improvement in aggregate benchmark scores

15.8% gain in security certification accuracy

Effective ablation studies validating dataset usefulness

Abstract

Large Language Models (LLMs) have shown remarkable advancements in specialized fields such as finance, law, and medicine. However, in cybersecurity, we have noticed a lack of open-source datasets, with a particular lack of high-quality cybersecurity pretraining corpora, even though much research indicates that LLMs acquire their knowledge during pretraining. To address this, we present a comprehensive suite of datasets covering all major training stages, including pretraining, instruction fine-tuning, and reasoning distillation with cybersecurity-specific self-reflection data. Extensive ablation studies demonstrate their effectiveness on public cybersecurity benchmarks. In particular, continual pre-training on our dataset yields a 15.9% improvement in the aggregate score, while reasoning distillation leads to a 15.8% gain in security certification (CISSP). We will release all datasets and trained cybersecurity LLMs under the ODC-BY and MIT licenses to encourage further research in the community. For access to all datasets and model weights, please refer to https://huggingface.co/collections/trendmicro-ailab/primus-67b1fd27052b802b4af9d243.