Accurate Human Body Reconstruction for Volumetric Video

TL;DR

This paper presents a deep learning-based approach to improve high-fidelity human body reconstruction in volumetric video using passive cameras, emphasizing enhanced depth estimation and post-processing techniques.

Contribution

It introduces a novel deep learning multi-view stereo network and a depth map post-processing method tailored for professional volumetric human body reconstruction.

Findings

Achieves high geometric detail in human body reconstructions.

Outperforms traditional stereo matching in volumetric video.

Enhances reconstruction quality with confidence-aware filtering.

Abstract

In this work, we enhance a professional end-to-end volumetric video production pipeline to achieve high-fidelity human body reconstruction using only passive cameras. While current volumetric video approaches estimate depth maps using traditional stereo matching techniques, we introduce and optimize deep learning-based multi-view stereo networks for depth map estimation in the context of professional volumetric video reconstruction. Furthermore, we propose a novel depth map post-processing approach including filtering and fusion, by taking into account photometric confidence, cross-view geometric consistency, foreground masks as well as camera viewing frustums. We show that our method can generate high levels of geometric detail for reconstructed human bodies.