Vibe Coding XR: Accelerating AI + XR Prototyping with XR Blocks and Gemini
Ruofei Du, Benjamin Hersh, David Li, Nels Numan, Xun Qian, Yanhe Chen, Zhongyi Zhou, Xingyue Chen, Jiahao Ren, Robert Timothy Bettridge, Xiang 'Anthony' Chen, Faraz Faruqi, Steve Toh, David Kim

TL;DR
This paper introduces XR Blocks, an open-source framework that simplifies prototyping XR experiences with AI by translating high-level prompts into functional applications, accelerating development.
Contribution
It presents XR Blocks, a semantic framework for spatial computing, and Vibe Coding XR, a workflow that enables rapid AI-driven XR prototyping from high-level prompts.
Findings
High success rate in one-shot execution of XR prompts
Enables rapid transition from idea to functional XR applications
Provides an open-source toolkit and evaluation dataset for XR prototyping.
Abstract
While large language models (LLMs) have accelerated 2D software development through intent-driven "vibe coding", prototyping intelligent Extended Reality (XR) experiences remains a major challenge. The fundamental barrier is not just the steep learning curve for human creators, but that low-level sensor APIs and complex game engine hierarchies are ill-suited for LLM reasoning, routinely exceeding context windows and inducing syntax hallucinations. To bridge this gap, we contribute XR Blocks, an open-source, LLM-native WebXR framework. Unlike traditional engines, XR Blocks introduces a semantic "Reality Model" that aligns spatial computing primitives (users, physical environments, and agents) with natural language, providing a robust, concise vocabulary optimized for generative AI. Building upon this foundation, we present Vibe Coding XR, an end-to-end prototyping workflow that leverages…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
