MUC: Mixture of Uncalibrated Cameras for Robust 3D Human Body Reconstruction

TL;DR

This paper presents a novel method for 3D human body reconstruction using multiple uncalibrated cameras, leveraging a neural network to weight views and integrating facial expressions and hand poses for detailed models.

Contribution

It introduces a new approach that reconstructs detailed 3D human models from uncalibrated multi-view data, handling self-occlusion and body shape continuity without camera calibration.

Findings

Effective reconstruction on public datasets

Supports flexible number of cameras

Enhanced detail with SMPL-X model

Abstract

Multiple cameras can provide comprehensive multi-view video coverage of a person. Fusing this multi-view data is crucial for tasks like behavioral analysis, although it traditionally requires camera calibration, a process that is often complex. Moreover, previous studies have overlooked the challenges posed by self-occlusion under multiple views and the continuity of human body shape estimation. In this study, we introduce a method to reconstruct the 3D human body from multiple uncalibrated camera views. Initially, we utilize a pre-trained human body encoder to process each camera view individually, enabling the reconstruction of human body models and parameters for each view along with predicted camera positions. Rather than merely averaging the models across views, we develop a neural network trained to assign weights to individual views for all human body joints, based on the estimated distribution of joint distances from each camera. Additionally, we focus on the mesh surface of the human body for dynamic fusion, allowing for the seamless integration of facial expressions and body shape into a unified human body model. Our method has shown excellent performance in reconstructing the human body on two public datasets, advancing beyond previous work from the SMPL model to the SMPL-X model. This extension incorporates more complex hand poses and facial expressions, enhancing the detail and accuracy of the reconstructions. Crucially, it supports the flexible ad-hoc deployment of any number of cameras, offering significant potential for various applications. Our code is available at https://github.com/AbsterZhu/MUC.