Identifying Smart Contract Security Issues in Code Snippets from Stack Overflow

TL;DR

This paper presents SOChecker, a novel tool that detects vulnerabilities in incomplete smart contract code snippets from Stack Overflow, combining code completion with symbolic analysis, and demonstrates its superior performance over GPT models.

Contribution

Introduces SOChecker, the first tool to identify vulnerabilities in incomplete SO smart contract snippets using a fine-tuned Llama2 model and symbolic execution, outperforming GPT-based approaches.

Findings

SOChecker achieves an F1 score of 68.2%.

Survey shows 86.4% of developers neglect security in reused code.

SOChecker outperforms GPT-3.5 and GPT-4 in vulnerability detection.

Abstract

Smart contract developers frequently seek solutions to developmental challenges on Q&A platforms such as Stack Overflow (SO). Although community responses often provide viable solutions, the embedded code snippets can also contain hidden vulnerabilities. Integrating such code directly into smart contracts may make them susceptible to malicious attacks. We conducted an online survey and received 74 responses from smart contract developers. The results of this survey indicate that the majority (86.4%) of participants do not sufficiently consider security when reusing SO code snippets. Despite the existence of various tools designed to detect vulnerabilities in smart contracts, these tools are typically developed for analyzing fully-completed smart contracts and thus are ineffective for analyzing typical code snippets as found on SO. We introduce SOChecker, the first tool designed to identify potential vulnerabilities in incomplete SO smart contract code snippets. SOChecker first leverages a fine-tuned Llama2 model for code completion, followed by the application of symbolic execution methods for vulnerability detection. Our experimental results, derived from a dataset comprising 897 code snippets collected from smart contract-related SO posts, demonstrate that SOChecker achieves an F1 score of 68.2%, greatly surpassing GPT-3.5 and GPT-4 (20.9% and 33.2% F1 Scores respectively). Our findings underscore the need to improve the security of code snippets from Q&A websites.