4D Human Body Correspondences from Panoramic Depth Maps

TL;DR

This paper introduces a deep learning framework that uses panoramic depth maps to establish dense human body shape correspondences and compress 3D data efficiently for free-viewpoint video applications.

Contribution

It presents a novel end-to-end method combining panoramic depth maps, feature learning, and autoencoder compression for 3D human body data.

Findings

Robustness on public and new datasets

Effective dense correspondence establishment

Significant data compression results

Abstract

The availability of affordable 3D full body reconstruction systems has given rise to free-viewpoint video (FVV) of human shapes. Most existing solutions produce temporally uncorrelated point clouds or meshes with unknown point/vertex correspondences. Individually compressing each frame is ineffective and still yields to ultra-large data sizes. We present an end-to-end deep learning scheme to establish dense shape correspondences and subsequently compress the data. Our approach uses sparse set of "panoramic" depth maps or PDMs, each emulating an inward-viewing concentric mosaics. We then develop a learning-based technique to learn pixel-wise feature descriptors on PDMs. The results are fed into an autoencoder-based network for compression. Comprehensive experiments demonstrate our solution is robust and effective on both public and our newly captured datasets.