Minerva: Reinforcement Learning with Verifiable Rewards for Cyber Threat Intelligence LLMs

TL;DR

Minerva introduces a reinforcement learning approach with verifiable rewards to improve large language models' performance in generating structured cyber threat intelligence outputs, leveraging community standards for deterministic verification.

Contribution

The paper presents Minerva, a novel dataset and training pipeline that uses verifiable rewards and self-training to enhance LLMs for CTI tasks, outperforming existing methods.

Findings

MinervaRL improves mean scores by 15.8 percentage points over base models.

The approach enhances output accuracy across multiple CTI benchmarks.

Verifiable rewards enable deterministic verification of structured outputs.

Abstract

Cyber threat intelligence (CTI) analysts routinely convert noisy, unstructured security artifacts into standardized, automation-ready representations. Although large language models (LLMs) show promise for this task, existing approaches remain brittle when producing structured CTI outputs and have largely relied on supervised fine-tuning (SFT). In contrast, CTI standards and community-maintained resources define canonical identifiers and schemas that enable deterministic verification of model outputs. We leverage this structure to study reinforcement learning with verifiable rewards (RLVR) for CTI tasks. We introduce Minerva, a unified dataset and training pipeline spanning multiple CTI subtasks, each paired with task-specific verifiers that score structured outputs and identifier predictions. To address reward sparsity during rollout, we propose MinervaRL, a lightweight self-training mechanism that generates additional verified trajectories and distills them back into the model. Averaged across four backbones and 12 CTI benchmarks, MinervaRL improves the mean score by 15.8 percentage points over the corresponding base models and by 4.3 points over GRPO.