EPOCH: Jointly Estimating the 3D Pose of Cameras and Humans

TL;DR

The paper introduces EPOCH, a framework that jointly estimates 3D human pose and camera parameters from a single image using a full perspective model, improving accuracy and generalization in monocular 3D pose estimation.

Contribution

EPOCH is the first to jointly estimate camera parameters and 3D human pose using a full perspective model with an unsupervised approach, enhancing accuracy and generalization.

Findings

Achieves state-of-the-art results on Human3.6M and MPI-INF-3DHP datasets.

Modeling the problem as an unsupervised task with camera in-the-loop improves generalization.

Utilizes only 2D pose data for weak supervision, reducing reliance on labeled 3D data.

Abstract

Monocular Human Pose Estimation (HPE) aims at determining the 3D positions of human joints from a single 2D image captured by a camera. However, a single 2D point in the image may correspond to multiple points in 3D space. Typically, the uniqueness of the 2D-3D relationship is approximated using an orthographic or weak-perspective camera model. In this study, instead of relying on approximations, we advocate for utilizing the full perspective camera model. This involves estimating camera parameters and establishing a precise, unambiguous 2D-3D relationship. To do so, we introduce the EPOCH framework, comprising two main components: the pose lifter network (LiftNet) and the pose regressor network (RegNet). LiftNet utilizes the full perspective camera model to precisely estimate the 3D pose in an unsupervised manner. It takes a 2D pose and camera parameters as inputs and produces the corresponding 3D pose estimation. These inputs are obtained from RegNet, which starts from a single image and provides estimates for the 2D pose and camera parameters. RegNet utilizes only 2D pose data as weak supervision. Internally, RegNet predicts a 3D pose, which is then projected to 2D using the estimated camera parameters. This process enables RegNet to establish the unambiguous 2D-3D relationship. Our experiments show that modeling the lifting as an unsupervised task with a camera in-the-loop results in better generalization to unseen data. We obtain state-of-the-art results for the 3D HPE on the Human3.6M and MPI-INF-3DHP datasets. Our code is available at: [Github link upon acceptance, see supplementary materials].