WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Games

TL;DR

WebGameBench is a novel benchmark that evaluates coding agents' ability to generate browser-native games from specifications, focusing on the actual application runtime and usability.

Contribution

It introduces WebGameBench, the first requirement-to-application benchmark for browser-native games, with a real browser runtime evaluator and human-aligned usability labels.

Findings

Best system achieves 76.9% usable rate but only 20.2% excellent.

WebGameBench effectively distinguishes current coding systems.

Crossing minimal playable threshold remains a challenge.

Abstract

Coding agents are increasingly used as application builders, yet many evaluations still focus on source code, repository-level tests, or intermediate traces rather than the delivered application. We introduce WebGameBench, a requirement-to-application benchmark that evaluates whether coding agents can turn a frozen Structured WebGame Specification into a browser-accessible game. Browser-native games provide a compact but behavior-dense testbed: even simple games require coordinated input handling, spatial mapping, rule execution, state transitions, terminal conditions, restart behavior, and visible feedback. In WebGameBench, each generated artifact is built, served, and exposed as a browser-accessible application under a unified deployment protocol. A runtime evaluator then interacts with the delivered game in a real browser and assigns a three-way label: EXCELLENT, USABLE, or UNUSABLE. On a human-reviewed subset, the runtime label is broadly aligned with human gameplay review under the Usable-rate criterion. Across 111 tasks, 12 coding agents, and 14 evaluation configurations, WebGameBench separates current systems: the best configuration reaches a 76.9% usable rate but only a 20.2% excellent rate. This gap shows that crossing the minimum playable-delivery threshold is still far from complete requirement satisfaction. To our knowledge, WebGameBench is the first requirement-to-application benchmark for browser-native game delivery that validates delivered-application runtime labels against independent human gameplay review under the Usable-rate criterion.