A Context-Driven Approach for Co-Auditing Smart Contracts with The Support of GPT-4 code interpreter

TL;DR

This paper presents a context-driven prompting technique for GPT-4 to improve smart contract co-auditing, significantly increasing vulnerability detection rates compared to standard prompting methods.

Contribution

The paper introduces a novel context-scoping prompting approach that enhances GPT-4's effectiveness in smart contract auditing by better managing input context and code segmentation.

Findings

Detection rate of 96% for vulnerable functions

Outperforms native prompting approach (53%)

Expert auditors confirm improved detection reliability

Abstract

The surge in the adoption of smart contracts necessitates rigorous auditing to ensure their security and reliability. Manual auditing, although comprehensive, is time-consuming and heavily reliant on the auditor's expertise. With the rise of Large Language Models (LLMs), there is growing interest in leveraging them to assist auditors in the auditing process (co-auditing). However, the effectiveness of LLMs in smart contract co-auditing is contingent upon the design of the input prompts, especially in terms of context description and code length. This paper introduces a novel context-driven prompting technique for smart contract co-auditing. Our approach employs three techniques for context scoping and augmentation, encompassing code scoping to chunk long code into self-contained code segments based on code inter-dependencies, assessment scoping to enhance context description based on the target assessment goal, thereby limiting the search space, and reporting scoping to force a specific format for the generated response. Through empirical evaluations on publicly available vulnerable contracts, our method demonstrated a detection rate of 96\% for vulnerable functions, outperforming the native prompting approach, which detected only 53\%. To assess the reliability of our prompting approach, manual analysis of the results was conducted by expert auditors from our partner, Quantstamp, a world-leading smart contract auditing company. The experts' analysis indicates that, in unlabeled datasets, our proposed approach enhances the proficiency of the GPT-4 code interpreter in detecting vulnerabilities.