Survey of Large Language Models in Extended Reality: Technical Paradigms and Application Frontiers

TL;DR

This survey reviews how Large Language Models are integrated into Extended Reality systems, highlighting technical paradigms and application opportunities across various domains to guide future research and development.

Contribution

It offers a structured taxonomy of LLM-enhanced XR systems, connecting technical paradigms with practical applications and identifying open challenges.

Findings

Identifies key technical paradigms like agent control and scene synthesis.

Highlights applications in education, healthcare, and manufacturing.

Discusses design considerations and future challenges.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation, and their integration with Extended Reality (XR) is poised to transform how users interact with immersive environments. This survey provides a comprehensive review of recent developments at the intersection of LLMs and XR, offering a structured organization of research along both technical and application dimensions. We propose a taxonomy of LLM-enhanced XR systems centered on key technical paradigms -- such as interactive agent control, XR development toolkits, and generative scene synthesis -- and discuss how these paradigms enable novel capabilities in XR. In parallel, we examine how LLM-driven techniques support practical XR applications across diverse domains, including immersive education, clinical healthcare, and industrial manufacturing. By connecting these technical paradigms with application frontiers, our survey highlights current trends, delineates design considerations, and identifies open challenges in building LLM-augmented XR systems. This work provides insights that can guide researchers and practitioners in advancing the state of the art in intelligent XR experiences.