An Empirical Study of AI-based Smart Contract Creation

TL;DR

This study systematically evaluates AI-generated smart contracts using LLMs, revealing security issues and quality concerns, and suggests directions for enhancing AI-assisted smart contract development.

Contribution

It provides a comprehensive evaluation framework for assessing LLM-generated smart contracts and highlights key security and quality challenges.

Findings

Security bugs are frequently introduced in generated contracts.

The overall correctness and quality of code are impacted by input parameters.

Identifies areas for improving AI-generated smart contract safety.

Abstract

The introduction of large language models (LLMs) like ChatGPT and Google Palm2 for smart contract generation seems to be the first well-established instance of an AI pair programmer. LLMs have access to a large number of open-source smart contracts, enabling them to utilize more extensive code in Solidity than other code generation tools. Although the initial and informal assessments of LLMs for smart contract generation are promising, a systematic evaluation is needed to explore the limits and benefits of these models. The main objective of this study is to assess the quality of generated code provided by LLMs for smart contracts. We also aim to evaluate the impact of the quality and variety of input parameters fed to LLMs. To achieve this aim, we created an experimental setup for evaluating the generated code in terms of validity, correctness, and efficiency. Our study finds crucial evidence of security bugs getting introduced in the generated smart contracts as well as the overall quality and correctness of the code getting impacted. However, we also identified the areas where it can be improved. The paper also proposes several potential research directions to improve the process, quality and safety of generated smart contract codes.