Mutual Scene Synthesis for Mixed Reality Telepresence

TL;DR

This paper introduces a novel method for synthesizing shared virtual scenes in mixed reality telepresence, enabling users to interact in a mutually accessible environment based on their physical spaces.

Contribution

It presents a new mutual scene synthesis approach combining optimization and deep learning to generate shared virtual environments from participants' real spaces.

Findings

Effective scene synthesis using MatterPort3D dataset

Participants preferred the synthesized scenes in user studies

The approach enables functional interactions like walking and sitting

Abstract

Remote telepresence via next-generation mixed reality platforms can provide higher levels of immersion for computer-mediated communications, allowing participants to engage in a wide spectrum of activities, previously not possible in 2D screen-based communication methods. However, as mixed reality experiences are limited to the local physical surrounding of each user, finding a common virtual ground where users can freely move and interact with each other is challenging. In this paper, we propose a novel mutual scene synthesis method that takes the participants' spaces as input, and generates a virtual synthetic scene that corresponds to the functional features of all participants' local spaces. Our method combines a mutual function optimization module with a deep-learning conditional scene augmentation process to generate a scene mutually and physically accessible to all participants of a mixed reality telepresence scenario. The synthesized scene can hold mutual walkable, sittable and workable functions, all corresponding to physical objects in the users' real environments. We perform experiments using the MatterPort3D dataset and conduct comparative user studies to evaluate the effectiveness of our system. Our results show that our proposed approach can be a promising research direction for facilitating contextualized telepresence systems for next-generation spatial computing platforms.