Real-Time Interactions Between Human Controllers and Remote Devices in Metaverse
Kan Chen, Zhen Meng, Xiangmin Xu, Changyang She, Philip G. Zhao

TL;DR
This paper introduces a novel framework for real-time human-controller interactions in the Metaverse, using joint motion prediction and reinforcement learning to reduce latency and improve synchronization with remote devices.
Contribution
It proposes a decoupled virtual model with dynamic prediction horizons and a two-step reinforcement learning approach to enhance real-time interaction accuracy and efficiency.
Findings
Reduces Motion-To-Photon latency significantly.
Decreases RMSE between human motion and remote device control.
Verifies effectiveness through prototype experiments with varying latencies.
Abstract
Supporting real-time interactions between human controllers and remote devices remains a challenging goal in the Metaverse due to the stringent requirements on computing workload, communication throughput, and round-trip latency. In this paper, we establish a novel framework for real-time interactions through the virtual models in the Metaverse. Specifically, we jointly predict the motion of the human controller for 1) proactive rendering in the Metaverse and 2) generating control commands to the real-world remote device in advance. The virtual model is decoupled into two components for rendering and control, respectively. To dynamically adjust the prediction horizons for rendering and control, we develop a two-step human-in-the-loop continuous reinforcement learning approach and use an expert policy to improve the training efficiency. An experimental prototype is built to verify our…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
