GameUIAgent: An LLM-Powered Framework for Automated Game UI Design with Structured Intermediate Representation

TL;DR

GameUIAgent leverages large language models and a structured intermediate representation to automate game UI design, improving consistency and efficiency through a neuro-symbolic pipeline with iterative self-correction.

Contribution

The paper introduces GameUIAgent, a novel LLM-powered framework that translates natural language into editable game UI designs using a structured JSON representation and a neuro-symbolic pipeline.

Findings

Identified a game-domain failure taxonomy highlighting rarity-dependent degradation and visual emptiness.

Discovered a Quality Ceiling Effect indicating limits of self-correction improvements.

Revealed that partial rendering can paradoxically reduce evaluation fidelity.

Abstract

Game UI design requires consistent visual assets across rarity tiers yet remains a predominantly manual process. We present GameUIAgent, an LLM-powered agentic framework that translates natural language descriptions into editable Figma designs via a Design Spec JSON intermediate representation. A six-stage neuro-symbolic pipeline combines LLM generation, deterministic post-processing, and a Vision-Language Model (VLM)-guided Reflection Controller (RC) for iterative self-correction with guaranteed non-regressive quality. Evaluated across 110 test cases, three LLMs, and three UI templates, cross-model analysis establishes a game-domain failure taxonomy (rarity-dependent degradation; visual emptiness) and uncovers two key empirical findings. A Quality Ceiling Effect (Pearson r=-0.96, p<0.01) suggests that RC improvement is bounded by headroom below a quality threshold -- a visual-domain counterpart to test-time compute scaling laws. A Rendering-Evaluation Fidelity Principle reveals that partial rendering enhancements paradoxically degrade VLM evaluation by amplifying structural defects. Together, these results establish foundational principles for LLM-driven visual generation agents in game production.