H3WB: Human3.6M 3D WholeBody Dataset and Benchmark

TL;DR

This paper introduces the H3WB dataset for 3D whole-body pose estimation, providing comprehensive annotations and benchmarks for various tasks, advancing the development of unified human pose models.

Contribution

The paper presents the first fully annotated 3D whole-body dataset, H3WB, with a multi-view pipeline, and establishes new benchmarks for unified human pose estimation tasks.

Findings

H3WB contains 133 keypoints on 100K images.

Automated annotations improve total capture dataset performance.

Baseline methods demonstrate the feasibility of the proposed tasks.

Abstract

We present a benchmark for 3D human whole-body pose estimation, which involves identifying accurate 3D keypoints on the entire human body, including face, hands, body, and feet. Currently, the lack of a fully annotated and accurate 3D whole-body dataset results in deep networks being trained separately on specific body parts, which are combined during inference. Or they rely on pseudo-groundtruth provided by parametric body models which are not as accurate as detection based methods. To overcome these issues, we introduce the Human3.6M 3D WholeBody (H3WB) dataset, which provides whole-body annotations for the Human3.6M dataset using the COCO Wholebody layout. H3WB comprises 133 whole-body keypoint annotations on 100K images, made possible by our new multi-view pipeline. We also propose three tasks: i) 3D whole-body pose lifting from 2D complete whole-body pose, ii) 3D whole-body pose lifting from 2D incomplete whole-body pose, and iii) 3D whole-body pose estimation from a single RGB image. Additionally, we report several baselines from popular methods for these tasks. Furthermore, we also provide automated 3D whole-body annotations of TotalCapture and experimentally show that when used with H3WB it helps to improve the performance. Code and dataset is available at https://github.com/wholebody3d/wholebody3d