SCRUBD: Smart Contracts Reentrancy and Unhandled Exceptions Vulnerability Dataset

TL;DR

SCRUBD provides a comprehensive dataset of real-world and synthesized Ethereum smart contracts labeled for reentrancy and unhandled exception vulnerabilities, enabling better evaluation of detection tools.

Contribution

This paper introduces SCRUBD, a standardized, labeled dataset for smart contract vulnerabilities, combining crowdsourced and expert-verified real-world data with synthesized scenarios.

Findings

Slither outperforms other tools on real-world datasets for RE and UX vulnerabilities.

Sailfish outperforms others on synthesized RE datasets.

SCRUBD enables effective comparison of vulnerability detection tools.

Abstract

Smart Contracts (SCs) handle transactions in the Ethereum blockchain worth millions of United States dollars, making them a lucrative target for attackers seeking to exploit vulnerabilities and steal funds. The Ethereum community has developed a rich set of tools to detect vulnerabilities in SCs, including reentrancy (RE) and unhandled exceptions (UX). A dataset of SCs labelled with vulnerabilities is needed to evaluate the tools' efficacy. Existing SC datasets with labelled vulnerabilities have limitations, such as covering only a limited range of vulnerability scenarios and containing incorrect labels. As a result, there is a lack of a standardized dataset to compare the performances of these tools. SCRUBD aims to fill this gap. We present a dataset of real-world SCs and synthesized SCs labelled with RE and UX. The real-world SC dataset is labelled through crowdsourcing, followed by manual inspection by an expert, and covers both RE and UX vulnerabilities. On the other hand, the synthesized dataset is carefully crafted to cover various RE scenarios only. Using SCRUBD we compared the performance of six popular vulnerability detection tools. Based on our study, we found that Slither outperforms other tools on a crowdsourced dataset in detecting RE vulnerabilities, while Sailfish outperforms other tools on a manually synthesized dataset for detecting RE. For UX vulnerabilities, Slither outperforms all other tools.