Joint Optimization for 4D Human-Scene Reconstruction in the Wild

TL;DR

This paper introduces JOSH, an optimization-based approach for 4D human-scene reconstruction from monocular videos in natural settings, improving accuracy in motion and scene modeling by joint optimization and a new efficient model, JOSH3R.

Contribution

The paper presents a novel joint optimization framework for 4D human-scene reconstruction in the wild and introduces JOSH3R, a more efficient model trained with pseudo-labels.

Findings

JOSH outperforms previous methods in human motion estimation and scene reconstruction.

JOSH3R achieves higher accuracy than optimization-free methods.

Joint optimization improves reconstruction quality.

Abstract

Reconstructing human motion and its surrounding environment is crucial for understanding human-scene interaction and predicting human movements in the scene. While much progress has been made in capturing human-scene interaction in constrained environments, those prior methods can hardly reconstruct the natural and diverse human motion and scene context from web videos. In this work, we propose JOSH, a novel optimization-based method for 4D human-scene reconstruction in the wild from monocular videos. JOSH uses techniques in both dense scene reconstruction and human mesh recovery as initialization, and then it leverages the human-scene contact constraints to jointly optimize the scene, the camera poses, and the human motion. Experiment results show JOSH achieves better results on both global human motion estimation and dense scene reconstruction by joint optimization of scene geometry and human motion. We further design a more efficient model, JOSH3R, and directly train it with pseudo-labels from web videos. JOSH3R outperforms other optimization-free methods by only training with labels predicted from JOSH, further demonstrating its accuracy and generalization ability.