FORGE: An LLM-driven Framework for Large-Scale Smart Contract Vulnerability Dataset Construction

TL;DR

FORGE introduces an automated, LLM-driven framework for constructing large-scale, high-quality smart contract vulnerability datasets, addressing manual annotation limitations and standardizing vulnerability classification.

Contribution

It is the first automated approach leveraging LLMs to extract and classify vulnerabilities from audit reports into CWE categories, improving dataset scale and consistency.

Findings

Generated dataset contains 81,390 Solidity files and 27,497 vulnerabilities.

High extraction precision of 95.6% and inter-rater agreement of 0.87.

Benchmarking reveals limitations of existing security tools.

Abstract

High-quality smart contract vulnerability datasets are critical for evaluating security tools and advancing smart contract security research. Two major limitations of current manual dataset construction are (1) labor-intensive and error-prone annotation processes limiting the scale, quality, and evolution of the dataset, and (2) absence of standardized classification rules results in inconsistent vulnerability categories and labeling results across different datasets. To address these limitations, we present FORGE, the first automated approach for constructing smart contract vulnerability datasets. FORGE leverages an LLM-driven pipeline to extract high-quality vulnerabilities from real-world audit reports and classify them according to the CWE, the most widely recognized classification in software security. FORGE employs a divide-and-conquer strategy to extract structured and self-contained vulnerability information from these reports. Additionally, it uses a tree-of-thoughts technique to classify the vulnerability information into the hierarchical CWE classification. To evaluate FORGE's effectiveness, we run FORGE on 6,454 real-world audit reports and generate a dataset comprising 81,390 solidity files and 27,497 vulnerability findings across 296 CWE categories. Manual assessment of the dataset demonstrates high extraction precision and classification consistency with human experts (precision of 95.6% and inter-rater agreement k-$\alpha$ of 0.87). We further validate the practicality of our dataset by benchmarking 13 existing security tools on our dataset. The results reveal the significant limitations in current detection capabilities. Furthermore, by analyzing the severity-frequency distribution patterns through a unified CWE perspective in our dataset, we highlight inconsistency between current smart contract research focus and priorities identified from real-world vulnerabilities...