Learning Dense Correspondence from Synthetic Environments

TL;DR

This paper introduces a method for training 2D-3D human shape and pose mapping models using synthetic data with known dense correspondences, improving generalization to real-world images.

Contribution

It demonstrates that synthetic data with precise ground truth can effectively train 2D-3D mapping models, reducing reliance on manual annotations.

Findings

Models trained on synthetic data perform well on real datasets.

Synthetic data enhances generalization across diverse conditions.

Training on synthetic data is a viable alternative to real data.

Abstract

Estimation of human shape and pose from a single image is a challenging task. It is an even more difficult problem to map the identified human shape onto a 3D human model. Existing methods map manually labelled human pixels in real 2D images onto the 3D surface, which is prone to human error, and the sparsity of available annotated data often leads to sub-optimal results. We propose to solve the problem of data scarcity by training 2D-3D human mapping algorithms using automatically generated synthetic data for which exact and dense 2D-3D correspondence is known. Such a learning strategy using synthetic environments has a high generalisation potential towards real-world data. Using different camera parameter variations, background and lighting settings, we created precise ground truth data that constitutes a wider distribution. We evaluate the performance of models trained on synthetic using the COCO dataset and validation framework. Results show that training 2D-3D mapping network models on synthetic data is a viable alternative to using real data.