Synthesizing Training Images for Boosting Human 3D Pose Estimation

TL;DR

This paper introduces a scalable method for synthesizing training images with ground truth 3D pose annotations to improve human 3D pose estimation, demonstrating that synthetic data can outperform real images when used for training CNNs.

Contribution

The authors propose a fully automatic approach to generate synthetic training images with pose annotations, emphasizing pose coverage and texture diversity, and explore domain adaptation techniques.

Findings

Synthetic training images improve 3D pose estimation accuracy.

Domain adaptation bridges the gap between synthetic and real images.

CNNs trained on synthetic data outperform those trained on real images.

Abstract

Human 3D pose estimation from a single image is a challenging task with numerous applications. Convolutional Neural Networks (CNNs) have recently achieved superior performance on the task of 2D pose estimation from a single image, by training on images with 2D annotations collected by crowd sourcing. This suggests that similar success could be achieved for direct estimation of 3D poses. However, 3D poses are much harder to annotate, and the lack of suitable annotated training images hinders attempts towards end-to-end solutions. To address this issue, we opt to automatically synthesize training images with ground truth pose annotations. Our work is a systematic study along this road. We find that pose space coverage and texture diversity are the key ingredients for the effectiveness of synthetic training data. We present a fully automatic, scalable approach that samples the human pose space for guiding the synthesis procedure and extracts clothing textures from real images. Furthermore, we explore domain adaptation for bridging the gap between our synthetic training images and real testing photos. We demonstrate that CNNs trained with our synthetic images out-perform those trained with real photos on 3D pose estimation tasks.