Data-Driven 3D Reconstruction of Dressed Humans From Sparse Views

TL;DR

This paper presents a novel data-driven method for reconstructing 3D dressed humans from sparse camera views, effectively handling depth ambiguities and occlusions by leveraging multi-view information through attention and local pattern encoding.

Contribution

It introduces an end-to-end approach with spatially consistent reconstruction, attention-based view fusion, and local 3D pattern encoding for improved 3D human reconstruction from few views.

Findings

Outperforms state-of-the-art methods quantitatively and qualitatively.

Achieves spatially consistent reconstructions in dynamic scenes.

Produces results comparable to multi-view stereo with fewer views.

Abstract

Recently, data-driven single-view reconstruction methods have shown great progress in modeling 3D dressed humans. However, such methods suffer heavily from depth ambiguities and occlusions inherent to single view inputs. In this paper, we tackle this problem by considering a small set of input views and investigate the best strategy to suitably exploit information from these views. We propose a data-driven end-to-end approach that reconstructs an implicit 3D representation of dressed humans from sparse camera views. Specifically, we introduce three key components: first a spatially consistent reconstruction that allows for arbitrary placement of the person in the input views using a perspective camera model; second an attention-based fusion layer that learns to aggregate visual information from several viewpoints; and third a mechanism that encodes local 3D patterns under the multi-view context. In the experiments, we show the proposed approach outperforms the state of the art on standard data both quantitatively and qualitatively. To demonstrate the spatially consistent reconstruction, we apply our approach to dynamic scenes. Additionally, we apply our method on real data acquired with a multi-camera platform and demonstrate our approach can obtain results comparable to multi-view stereo with dramatically less views.