Vibe Coding XR: Accelerating AI + XR Prototyping with XR Blocks and Gemini

TL;DR

This paper introduces XR Blocks, an open-source framework that simplifies prototyping XR experiences with AI by translating high-level prompts into functional applications, accelerating development.

Contribution

It presents XR Blocks, a semantic framework for spatial computing, and Vibe Coding XR, a workflow that enables rapid AI-driven XR prototyping from high-level prompts.

Findings

High success rate in one-shot execution of XR prompts

Enables rapid transition from idea to functional XR applications

Provides an open-source toolkit and evaluation dataset for XR prototyping.

Abstract

While large language models (LLMs) have accelerated 2D software development through intent-driven "vibe coding", prototyping intelligent Extended Reality (XR) experiences remains a major challenge. The fundamental barrier is not just the steep learning curve for human creators, but that low-level sensor APIs and complex game engine hierarchies are ill-suited for LLM reasoning, routinely exceeding context windows and inducing syntax hallucinations. To bridge this gap, we contribute XR Blocks, an open-source, LLM-native WebXR framework. Unlike traditional engines, XR Blocks introduces a semantic "Reality Model" that aligns spatial computing primitives (users, physical environments, and agents) with natural language, providing a robust, concise vocabulary optimized for generative AI. Building upon this foundation, we present Vibe Coding XR, an end-to-end prototyping workflow that leverages LLMs to translate high-level prompts (e.g., "create a dandelion that reacts to my hand") directly into functional, physics-aware mixed-reality applications. To minimize the friction of on-device testing, the workflow introduces a seamless desktop "simulated reality" to headset deployment loop. Finally, we introduce VCXR60, a pilot dataset of 60 XR prompts paired with an automated evaluation pipeline. Our technical evaluation demonstrates high one-shot execution success, enabling practitioners to bypass lowlevel hurdles and rapidly move from "idea to reality". Code and live demos are available at https://github.com/google/xrblocks and http://xrblocks.github.io/gem.