Pipeline for 3D reconstruction of the human body from AR/VR headset mounted egocentric cameras

TL;DR

This paper introduces a pipeline that uses egocentric cameras and generative models to reconstruct and animate full 3D human body meshes, enabling applications like telepresence and virtual reality interactions.

Contribution

The novel pipeline combines egocentric view translation with 3D reconstruction and texturing, advancing full-body 3D modeling from limited viewpoints.

Findings

Achieves realistic 3D human body meshes from egocentric views.

Enables real-time animation and pose transfer.

Improves occlusion handling with view translation.

Abstract

In this paper, we propose a novel pipeline for the 3D reconstruction of the full body from egocentric viewpoints. 3-D reconstruction of the human body from egocentric viewpoints is a challenging task as the view is skewed and the body parts farther from the cameras are occluded. One such example is the view from cameras installed below VR headsets. To achieve this task, we first make use of conditional GANs to translate the egocentric views to full body third-person views. This increases the comprehensibility of the image and caters to occlusions. The generated third-person view is further sent through the 3D reconstruction module that generates a 3D mesh of the body. We also train a network that can take the third person full-body view of the subject and generate the texture maps for applying on the mesh. The generated mesh has fairly realistic body proportions and is fully rigged allowing for further applications such as real-time animation and pose transfer in games. This approach can be key to a new domain of mobile human telepresence.