When Generative AI Meets Extended Reality: Enabling Scalable and Natural Interactions

TL;DR

This paper explores integrating Generative AI with Extended Reality to enable scalable, natural interactions by automating content creation and interpreting intuitive language commands, thus reducing barriers to widespread XR adoption.

Contribution

It introduces novel methods for combining GenAI with XR, demonstrating practical use cases that enhance scalability and natural interaction in immersive environments.

Findings

GenAI can interpret ambiguous instructions in XR environments.

Automated 3D content generation lowers content creation barriers.

Integration improves user experience and interaction naturalness.

Abstract

Extended Reality (XR), including virtual, augmented, and mixed reality, provides immersive and interactive experiences across diverse applications, from VR-based education to AR-based assistance and MR-based training. However, widespread XR adoption remains limited due to two key challenges: 1) the high cost and complexity of authoring 3D content, especially for large-scale environments or complex interactions; and 2) the steep learning curve associated with non-intuitive interaction methods like handheld controllers or scripted gestures. Generative AI (GenAI) presents a promising solution by enabling intuitive, language-driven interaction and automating content generation. Leveraging vision-language models and diffusion-based generation, GenAI can interpret ambiguous instructions, understand physical scenes, and generate or manipulate 3D content, significantly lowering barriers to XR adoption. This paper explores the integration of XR and GenAI through three concrete use cases, showing how they address key obstacles in scalability and natural interaction, and identifying technical challenges that must be resolved to enable broader adoption.