Adaptive Hierarchical Evaluation of LLMs and SAST tools for CWE Prediction in Python

TL;DR

This paper introduces ALPHA, a hierarchical, CWE-specific benchmark for evaluating LLMs and SAST tools at the function level in Python, emphasizing diagnostic utility and prediction consistency.

Contribution

It presents the first function-level CWE-specific benchmark with hierarchical penalties, enabling more nuanced evaluation of vulnerability detection tools.

Findings

LLMs outperform SAST tools in vulnerability detection.

SAST tools have higher precision when detections are made.

Prediction consistency varies widely across models.

Abstract

Large Language Models have become integral to software development, yet they frequently generate vulnerable code. Existing code vulnerability detection benchmarks employ binary classification, lacking the CWE-level specificity required for actionable feedback in iterative correction systems. We present ALPHA (Adaptive Learning via Penalty in Hierarchical Assessment), the first function-level Python benchmark that evaluates both LLMs and SAST tools using hierarchically aware, CWE-specific penalties. ALPHA distinguishes between over-generalisation, over-specification, and lateral errors, reflecting practical differences in diagnostic utility. Evaluating seven LLMs and two SAST tools, we find LLMs substantially outperform SAST, though SAST demonstrates higher precision when detections occur. Critically, prediction consistency varies dramatically across models (8.26%-81.87% agreement), with significant implications for feedback-driven systems. We further outline a pathway for future work incorporating ALPHA penalties into supervised fine-tuning, which could provide principled hierarchy-aware vulnerability detection pending empirical validation.