Multi-hypotheses Conditioned Point Cloud Diffusion for 3D Human Reconstruction from Occluded Images

TL;DR

This paper introduces MHCDIFF, a novel point cloud diffusion approach conditioned on multiple hypotheses for detailed 3D human reconstruction from occluded images, outperforming existing methods in handling occlusions and misalignments.

Contribution

The paper proposes a multi-hypotheses conditioned point cloud diffusion model that improves 3D human reconstruction under occlusion by capturing global features and correcting misaligned meshes.

Findings

Outperforms state-of-the-art methods on CAPE and MultiHuman datasets.

Effectively handles severe occlusions and misalignments.

Generates detailed and consistent 3D human shapes from occluded images.

Abstract

3D human shape reconstruction under severe occlusion due to human-object or human-human interaction is a challenging problem. Parametric models i.e., SMPL(-X), which are based on the statistics across human shapes, can represent whole human body shapes but are limited to minimally-clothed human shapes. Implicit-function-based methods extract features from the parametric models to employ prior knowledge of human bodies and can capture geometric details such as clothing and hair. However, they often struggle to handle misaligned parametric models and inpaint occluded regions given a single RGB image. In this work, we propose a novel pipeline, MHCDIFF, Multi-hypotheses Conditioned Point Cloud Diffusion, composed of point cloud diffusion conditioned on probabilistic distributions for pixel-aligned detailed 3D human reconstruction under occlusion. Compared to previous implicit-function-based methods, the point cloud diffusion model can capture the global consistent features to generate the occluded regions, and the denoising process corrects the misaligned SMPL meshes. The core of MHCDIFF is extracting local features from multiple hypothesized SMPL(-X) meshes and aggregating the set of features to condition the diffusion model. In the experiments on CAPE and MultiHuman datasets, the proposed method outperforms various SOTA methods based on SMPL, implicit functions, point cloud diffusion, and their combined, under synthetic and real occlusions. Our code is publicly available at https://donghwankim0101.github.io/projects/mhcdiff/ .