UNOC: Understanding Occlusion for Embodied Presence in Virtual Reality

TL;DR

This paper introduces UNOC, a data-driven framework that improves inside-out body tracking in VR by addressing occlusion challenges through a new large-scale dataset and deep learning inference, enhancing real-time embodied presence.

Contribution

The paper presents a novel large-scale motion capture dataset and a deep learning approach for occlusion-aware inside-out body tracking in VR environments.

Findings

Effective inference of occluded body parts in real-time

High-fidelity pose generation during social interactions

Improved accuracy over traditional inverse kinematics methods

Abstract

Tracking body and hand motions in the 3D space is essential for social and self-presence in augmented and virtual environments. Unlike the popular 3D pose estimation setting, the problem is often formulated as inside-out tracking based on embodied perception (e.g., egocentric cameras, handheld sensors). In this paper, we propose a new data-driven framework for inside-out body tracking, targeting challenges of omnipresent occlusions in optimization-based methods (e.g., inverse kinematics solvers). We first collect a large-scale motion capture dataset with both body and finger motions using optical markers and inertial sensors. This dataset focuses on social scenarios and captures ground truth poses under self-occlusions and body-hand interactions. We then simulate the occlusion patterns in head-mounted camera views on the captured ground truth using a ray casting algorithm and learn a deep neural network to infer the occluded body parts. In the experiments, we show that our method is able to generate high-fidelity embodied poses by applying the proposed method on the task of real-time inside-out body tracking, finger motion synthesis, and 3-point inverse kinematics.