NoCode-bench: A Benchmark for Evaluating Natural Language-Driven Feature Addition

TL;DR

NoCode-bench is a new benchmark for evaluating large language models on real-world natural language-driven feature addition tasks, revealing current limitations of LLMs in no-code development.

Contribution

Introduces NoCode-bench, a comprehensive benchmark with 634 tasks for assessing LLMs in natural language-driven software development, including a verified subset for reliable evaluation.

Findings

Best LLMs achieve only 28.07% success rate

Challenges include cross-file editing and codebase understanding

LLMs are not yet ready for fully NL-driven no-code development

Abstract

Natural language-driven no-code development allows users to specify software functionality using natural language (NL) instead of editing source code, promising increased productivity and democratized development. Large language models (LLMs) show potential in enabling this paradigm. In this context, software documentation acts as an NL specification for functionality. This work introduces NoCode-bench, a benchmark designed to evaluate LLMs on real-world NL-driven feature addition tasks, consisting of 634 tasks across 10 projects and 114k code changes. Each task pairs documentation updates with corresponding code implementations, validated by developer-written test cases. A subset of 114 high-quality, human-verified instances, NoCode-bench Verified, ensures reliable evaluation. Our experiments reveal that, despite high token usage, the best LLMs achieve a task success rate of only 28.07%, highlighting challenges in cross-file editing, codebase understanding, and tool calling. These findings indicate that LLMs are not yet ready for fully NL-driven no-code development. NoCode-bench lays the foundation for future advances in this area.