LIVE-GS: LLM Powers Interactive VR Experience with Physics-Aware Gaussian Splatting

TL;DR

LIVE-GS is a VR system that uses Large Language Models to rapidly generate and simulate physics-based dynamic assets from static Gaussian objects, enabling realistic interactions in real-time.

Contribution

The paper introduces a novel LLM-powered VR system that predicts physical parameters from static assets in seconds, simplifying dynamic asset creation for non-experts.

Findings

LIVE-GS predicts physical parameters in 10 seconds with high accuracy.

User studies show improved efficiency and satisfaction in VR scene creation.

LIVE-GS enables realistic physics interactions without manual annotation.

Abstract

As 3D Gaussian Splatting (3DGS) emerges as a leading approach for novel view synthesis and scene reconstruction, its potential in digital asset creation has gained significant attention. An increasing number of asset libraries based on GS are being established. However, generating physics-based dynamic assets remains a time-consuming and expertise-intensive task, especially for non-experts. In this paper, we propose LIVE-GS, a highly realistic Virtual Reality (VR) system powered by Large Language Models (LLMs), which enables rapid creation of dynamic Gaussian assets and real-time VR interactions. To inform our system design, we conducted interviews to examine challenges faced by current GS-based VR systems and the specific demands of users. Based on these insights, we employed GPT-4o to analyze key physical properties of objects that significantly impact user interactions, ensuring physics-based interactions in VR align with real-world phenomena. A key innovation of LIVE-GS is its ability to predict reasonable parameters in just 10 seconds from static Gaussian assets while maintaining high-quality VR interactions. To validate our approach, we invited participants experienced in physical simulation to manually adjust physical parameters, providing a baseline for comparison in both asset quality and authoring efficiency. We also conducted a comprehensive user study to evaluate system usability and user satisfaction. Experimental results demonstrate that LIVE-GS, leveraging LLMs' scene understanding capabilities, can achieve efficient physical scene creation and natural interactions without requiring manual design or annotation.