From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detection

TL;DR

This paper systematically investigates post-training techniques for LLM-based vulnerability detection, demonstrating on-policy RL with GRPO outperforms other methods and providing new guidelines for data curation, training stages, rewards, and evaluation.

Contribution

It is the first comprehensive study applying post-training pipelines to vulnerability detection, revealing effective strategies and insights beyond common practices.

Findings

On-policy RL with GRPO outperforms SFT and preference optimization.

Rejection sampling-based SFT is more effective than rationalization supervision.

Root-cause analysis-based evaluation offers more robust assessment.

Abstract

The integration of LLMs into vulnerability detection (VD) has shifted the field toward more interpretable and context-aware analysis. While post-training techniques have shown promise in general coding tasks, their systematic application to VD remains underexplored. In this paper, we present the first comprehensive investigation into the post-training pipeline for LLM-based VD, demonstrating that on-policy RL with GRPO consistently outperforms SFT, off-policy preference optimization methods, and specialized VD LLMs. Our study further reveals VD-specific post-training guidelines and insights beyond common practices: (1) For data curation, contrary to the widespread use of rationalization-based supervision in prior VD work, SFT based on rejection sampling proves more effective, as rationalization can introduce hallucinations; in RL training, the inherently skewed difficulty distribution of vulnerabilities leads difficulty-aware data filtering to drastically reduce data coverage, causing non-negligible performance loss, and undermines curriculum learning, while pair-based data scheduling can partially mitigate this. (2) For stage interactions, unlike preference optimization typically applied to lightly trained SFT models, increasing SFT epochs consistently benefits off-policy preference optimization in VD tasks; however, excessive SFT suppresses self-exploration in on-policy RL, limiting its gains. (3) For reward mechanisms, naively treating vulnerability classification correctness as reward signals leads to reward hacking, whereas fine-grained root-cause judgments provide more reliable credit assignment; specification-based rewards further improve efficiency at the cost of additional design and generation effort. (4) For evaluation protocols, LLM-as-a-Judge based on root-cause analysis offers a more robust alternative, albeit with variability across judge models.