On LLM-Assisted Generation of Smart Contracts from Business Processes

TL;DR

This paper explores the use of large language models for generating smart contract code from business process descriptions, introducing an automated evaluation framework and empirical analysis of LLM capabilities.

Contribution

It presents a novel benchmarking framework and empirical data assessing LLM performance in generating reliable smart contract code from process models.

Findings

LLMs often fail to reliably enforce process flow and data conditions.

Performance varies significantly across different LLM types and sizes.

Current LLMs are not yet suitable for fully autonomous smart contract generation.

Abstract

Large language models (LLMs) have changed the reality of how software is produced. Within the wider software engineering community, among many other purposes, they are explored for code generation use cases from different types of input. In this work, we present an exploratory study to investigate the use of LLMs for generating smart contract code from business process descriptions, an idea that has emerged in recent literature to overcome the limitations of traditional rule-based code generation approaches. However, current LLM-based work evaluates generated code on small samples, relying on manual inspection, or testing whether code compiles but ignoring correct execution. With this work, we introduce an automated evaluation framework and provide empirical data from larger data sets of process models. We test LLMs of different types and sizes in their capabilities of achieving important properties of process execution, including enforcing process flow, resource allocation, and data-based conditions. Our results show that LLM performance falls short of the perfect reliability required for smart contract development. We suggest future work to explore responsible LLM integrations in existing tools for code generation to ensure more reliable output. Our benchmarking framework can serve as a foundation for developing and evaluating such integrations.