Humans as a Calibration Pattern: Dynamic 3D Scene Reconstruction from Unsynchronized and Uncalibrated Videos

TL;DR

This paper presents a method for reconstructing dynamic 3D scenes from unsynchronized, uncalibrated videos of humans by estimating shape, pose, time offsets, and camera poses, enabling accurate scene reconstruction without synchronized inputs.

Contribution

It introduces a novel approach to calibrate and reconstruct dynamic scenes from unsynchronized, uncalibrated videos using human shape and pose estimation as a calibration pattern.

Findings

Achieves accurate spatio-temporal calibration in challenging conditions

Reconstructs high-quality dynamic 3D scenes from unsynchronized videos

Refines time offsets and camera poses concurrently during training

Abstract

Recent works on dynamic 3D neural field reconstruction assume the input from synchronized multi-view videos whose poses are known. The input constraints are often not satisfied in real-world setups, making the approach impractical. We show that unsynchronized videos from unknown poses can generate dynamic neural fields as long as the videos capture human motion. Humans are one of the most common dynamic subjects captured in videos, and their shapes and poses can be estimated using state-of-the-art libraries. While noisy, the estimated human shape and pose parameters provide a decent initialization point to start the highly non-convex and under-constrained problem of training a consistent dynamic neural representation. Given the shape and pose parameters of humans in individual frames, we formulate methods to calculate the time offsets between videos, followed by camera pose estimations that analyze the 3D joint positions. Then, we train the dynamic neural fields employing multiresolution grids while we concurrently refine both time offsets and camera poses. The setup still involves optimizing many parameters; therefore, we introduce a robust progressive learning strategy to stabilize the process. Experiments show that our approach achieves accurate spatio-temporal calibration and high-quality scene reconstruction in challenging conditions.