Simultaneously Recovering Multi-Person Meshes and Multi-View Cameras with Human Semantics

TL;DR

This paper presents a novel method for multi-person motion capture using uncalibrated multi-view cameras, leveraging human semantics and motion priors to accurately recover human meshes and camera parameters without calibration tools.

Contribution

It introduces a unified approach that estimates camera parameters and human meshes simultaneously from noisy human semantics, eliminating the need for calibration tools or background features.

Findings

Accurate camera parameters and human motions achieved in one-step reconstruction.

Method effectively handles occlusions and inter-person interactions.

No reliance on background features or calibration procedures.

Abstract

Dynamic multi-person mesh recovery has broad applications in sports broadcasting, virtual reality, and video games. However, current multi-view frameworks rely on a time-consuming camera calibration procedure. In this work, we focus on multi-person motion capture with uncalibrated cameras, which mainly faces two challenges: one is that inter-person interactions and occlusions introduce inherent ambiguities for both camera calibration and motion capture; the other is that a lack of dense correspondences can be used to constrain sparse camera geometries in a dynamic multi-person scene. Our key idea is to incorporate motion prior knowledge to simultaneously estimate camera parameters and human meshes from noisy human semantics. We first utilize human information from 2D images to initialize intrinsic and extrinsic parameters. Thus, the approach does not rely on any other calibration tools or background features. Then, a pose-geometry consistency is introduced to associate the detected humans from different views. Finally, a latent motion prior is proposed to refine the camera parameters and human motions. Experimental results show that accurate camera parameters and human motions can be obtained through a one-step reconstruction. The code are publicly available at~\url{https://github.com/boycehbz/DMMR}.