Holistic 3D Human and Scene Mesh Estimation from Single View Images

TL;DR

This paper introduces an end-to-end model that jointly estimates 3D human and scene meshes from a single image, outperforming existing methods by leveraging comprehensive loss functions and joint optimization.

Contribution

It is the first model to simultaneously predict human and object meshes and optimize scene and human poses together from a single RGB image.

Findings

Outperforms existing human mesh estimation methods.

Outperforms existing indoor scene reconstruction methods.

First to jointly estimate human and object meshes at the mesh level.

Abstract

The 3D world limits the human body pose and the human body pose conveys information about the surrounding objects. Indeed, from a single image of a person placed in an indoor scene, we as humans are adept at resolving ambiguities of the human pose and room layout through our knowledge of the physical laws and prior perception of the plausible object and human poses. However, few computer vision models fully leverage this fact. In this work, we propose an end-to-end trainable model that perceives the 3D scene from a single RGB image, estimates the camera pose and the room layout, and reconstructs both human body and object meshes. By imposing a set of comprehensive and sophisticated losses on all aspects of the estimations, we show that our model outperforms existing human body mesh methods and indoor scene reconstruction methods. To the best of our knowledge, this is the first model that outputs both object and human predictions at the mesh level, and performs joint optimization on the scene and human poses.