FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluation

TL;DR

FrontendBench is a comprehensive, interactive benchmark with automatic evaluation for assessing large language models' ability to generate realistic front-end code, addressing limitations of previous simplistic and non-rigorous tests.

Contribution

We introduce FrontendBench, a new benchmark with interactive test scenarios and an automatic evaluation framework for more accurate assessment of LLMs in front-end development.

Findings

High agreement (90.54%) between automatic and human evaluations.

Significant performance disparities among state-of-the-art LLMs.

Benchmark covers diverse web components and realistic development challenges.

Abstract

Large Language Models (LLMs) have made significant strides in front-end code generation. However, existing benchmarks exhibit several critical limitations: many tasks are overly simplistic, test cases often lack rigor, and end-to-end validation is absent. These issues hinder the accurate assessment of model performance. To address these challenges, we present FrontendBench, a benchmark co-developed by humans and LLMs. FrontendBench categorizes tasks based on code functionality and incorporates interactive test scenarios, enabling a more comprehensive and practical evaluation of front-end code generation capabilities. The benchmark comprises 148 meticulously crafted prompt-test case pairs spanning five levels of web components, from basic UI elements to complex interactive features. Each task reflects realistic front-end development challenges. Furthermore, we introduce an automatic evaluation framework that executes generated code within a sandbox environment and assesses outcomes using predefined test scripts. This framework achieves a 90.54% agreement rate with expert human evaluations, demonstrating high reliability. We benchmark several state-of-the-art LLMs on FrontendBench and observe substantial performance disparities in handling real-world front-end tasks. These results highlight FrontendBench as a reliable and scalable benchmark, supporting consistent multimodal evaluation and providing a robust foundation for future research in front-end code generation. Our data and code will be released soon.