Large Model Empowered Metaverse: State-of-the-Art, Challenges and Opportunities

TL;DR

This paper explores how large AI models can enhance the Metaverse by improving interaction, content creation, and rendering efficiency, while addressing scalability and responsiveness challenges through a novel AI-driven rendering framework.

Contribution

It introduces a generative AI-based framework with cloud-edge-end collaboration, mobility-aware pre-rendering, and diffusion model strategies to optimize Metaverse rendering performance.

Findings

Enhanced rendering efficiency demonstrated in experiments.

Reduced rendering overheads with the proposed framework.

Improved responsiveness and immersion in the Metaverse environment.

Abstract

The Metaverse represents a transformative shift beyond traditional mobile Internet, creating an immersive, persistent digital ecosystem where users can interact, socialize, and work within 3D virtual environments. Powered by large models such as ChatGPT and Sora, the Metaverse benefits from precise large-scale real-world modeling, automated multimodal content generation, realistic avatars, and seamless natural language understanding, which enhance user engagement and enable more personalized, intuitive interactions. However, challenges remain, including limited scalability, constrained responsiveness, and low adaptability in dynamic environments. This paper investigates the integration of large models within the Metaverse, examining their roles in enhancing user interaction, perception, content creation, and service quality. To address existing challenges, we propose a generative AI-based framework for optimizing Metaverse rendering. This framework includes a cloud-edge-end collaborative model to allocate rendering tasks with minimal latency, a mobility-aware pre-rendering mechanism that dynamically adjusts to user movement, and a diffusion model-based adaptive rendering strategy to fine-tune visual details. Experimental results demonstrate the effectiveness of our approach in enhancing rendering efficiency and reducing rendering overheads, advancing large model deployment for a more responsive and immersive Metaverse.