WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Quality

TL;DR

WebDevJudge introduces a benchmark for evaluating the reliability of LLMs as judges in web development, revealing significant gaps compared to human experts and highlighting key limitations of current models.

Contribution

This work provides the first systematic benchmark for assessing LLM-based evaluation in web development, including both static and interactive scenarios.

Findings

LLMs significantly lag behind human experts in web development evaluation.

Fundamental model limitations include failure to recognize functional equivalence and verify task feasibility.

The benchmark reveals critical areas for improving LLM evaluation capabilities.

Abstract

The paradigm of LLM-as-a-judge is emerging as a scalable and efficient alternative to human evaluation, demonstrating strong performance on well-defined tasks. However, its reliability in open-ended tasks with dynamic environments and complex interactions remains unexplored. To bridge the gap, we introduce WebDevJudge, a systematic benchmark for assessing LLM-as-a-judge performance in web development, with support for both non-interactive evaluation based on static observations and continuous interactive evaluation with a dynamic web environment. WebDevJudge comprises human preference labels over paired web implementations, annotated with structured and query-grounded rubrics to ensure high-quality ground truth. Using this benchmark, we comprehensively evaluate various evaluators, including LLMs, MLLMs, and agentic workflows. We systematically investigate the impact of different paradigms and guidance mechanisms. Our experiments reveal a significant gap between LLM judges and human experts. In-depth analysis indicates this gap stems from fundamental model limitations, including failures in recognizing functional equivalence, verifying task feasibility, and mitigating bias. Overall, WebDevJudge presents a challenge to LLM-as-a-judge, offering insights to guide future research toward developing more reliable and capable automated evaluators for complicated scenarios. Code and data are available at https://github.com/lcy2723/WebDevJudge.