An end-to-end agentic pipeline for smart contract translation and quality evaluation

TL;DR

This paper introduces an end-to-end framework that translates natural language into smart contracts, evaluates their quality comprehensively, and provides a reproducible benchmark for assessing synthesis accuracy and identifying systematic errors.

Contribution

The framework combines structured parsing, code generation, and multi-dimensional quality assessment using agent teams with iterative refinement, advancing systematic evaluation of LLM-generated smart contracts.

Findings

Framework effectively measures multiple quality dimensions.

Supports paired evaluation against ground-truth implementations.

Identifies systematic error modes such as logic omissions.

Abstract

We present an end-to-end framework for systematic evaluation of LLM-generated smart contracts from natural-language specifications. The system parses contractual text into structured schemas, generates Solidity code, and performs automated quality assessment through compilation and security checks. Using CrewAI-style agent teams with iterative refinement, the pipeline produces structured artifacts with full provenance metadata. Quality is measured across five dimensions, including functional completeness, variable fidelity, state-machine correctness, business-logic fidelity, and code quality aggregated into composite scores. The framework supports paired evaluation against ground-truth implementations, quantifying alignment and identifying systematic error modes such as logic omissions and state transition inconsistencies. This provides a reproducible benchmark for empirical research on smart contract synthesis quality and supports extensions to formal verification and compliance checking.