Legal Compliance Evaluation of Smart Contracts Generated By Large Language Models

TL;DR

This paper explores the potential of large language models to generate legally compliant smart contracts directly from natural language legal texts, proposing new metrics to evaluate legal compliance and analyzing their effectiveness.

Contribution

It introduces a novel set of metrics for assessing legal compliance in smart contracts generated by LLMs and evaluates their effectiveness across different models and legal contracts.

Findings

Larger LLMs tend to produce more legally compliant smart contracts.

The proposed metrics effectively distinguish levels of legal compliance.

LLMs can generate initial code for compliant smart contracts, aiding automated development.

Abstract

Smart contracts can implement and automate parts of legal contracts, but ensuring their legal compliance remains challenging. Existing approaches such as formal specification, verification, and model-based development require expertise in both legal and software development domains, as well as extensive manual effort. Given the recent advances of Large Language Models (LLMs) in code generation, we investigate their ability to generate legally compliant smart contracts directly from natural language legal contracts, addressing these challenges. We propose a novel suite of metrics to quantify legal compliance based on modeling both legal and smart contracts as processes and comparing their behaviors. We select four LLMs, generate 20 smart contracts based on five legal contracts, and analyze their legal compliance. We find that while all LLMs generate syntactically correct code, there is significant variance in their legal compliance with larger models generally showing higher levels of compliance. We also evaluate the proposed metrics against properties of software metrics, showing they provide fine-grained distinctions, enable nuanced comparisons, and are applicable across domains for code from any source, LLM or developer. Our results suggest that LLMs can assist in generating starter code for legally compliant smart contracts with strict reviews, and the proposed metrics provide a foundation for automated and self-improving development workflows.