ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents

TL;DR

ScreenCoder introduces a modular multi-agent system for transforming UI images into front-end code, significantly improving accuracy and robustness over monolithic models through specialized stages and scalable data generation.

Contribution

It proposes a novel multi-agent framework that decomposes UI-to-code tasks into interpretable stages, enhancing robustness and enabling scalable data creation for model fine-tuning.

Findings

Achieves state-of-the-art layout accuracy and code correctness.

Significantly improves robustness over end-to-end models.

Enables scalable high-quality data generation for training.

Abstract

Automating the transformation of user interface (UI) designs into front-end code holds significant promise for accelerating software development and democratizing design workflows. While multimodal large language models (MLLMs) can translate images to code, they often fail on complex UIs, struggling to unify visual perception, layout planning, and code synthesis within a single monolithic model, which leads to frequent perception and planning errors. To address this, we propose ScreenCoder, a modular multi-agent framework that decomposes the task into three interpretable stages: grounding, planning, and generation. By assigning these distinct responsibilities to specialized agents, our framework achieves significantly higher robustness and fidelity than end-to-end approaches. Furthermore, ScreenCoder serves as a scalable data engine, enabling us to generate high-quality image-code pairs. We use this data to fine-tune open-source MLLM via a dual-stage pipeline of supervised fine-tuning and reinforcement learning, demonstrating substantial gains in its UI generation capabilities. Extensive experiments demonstrate that our approach achieves state-of-the-art performance in layout accuracy, structural coherence, and code correctness. Our code is made publicly available at https://github.com/leigest519/ScreenCoder.