CanonicalFusion: Generating Drivable 3D Human Avatars from Multiple Images

TL;DR

CanonicalFusion is a new framework that reconstructs animatable 3D human avatars from multiple images by integrating individual reconstructions into a canonical space, using a novel skinning and differentiable rendering approach.

Contribution

It introduces a method to predict compressed skinning weights and employs a forward skinning-based differentiable rendering scheme for improved 3D human avatar reconstruction.

Findings

Effective reconstruction of drivable 3D human avatars from multiple images.

Outperforms state-of-the-art methods in accuracy and robustness.

Open-source implementation available for reproducibility.

Abstract

We present a novel framework for reconstructing animatable human avatars from multiple images, termed CanonicalFusion. Our central concept involves integrating individual reconstruction results into the canonical space. To be specific, we first predict Linear Blend Skinning (LBS) weight maps and depth maps using a shared-encoder-dual-decoder network, enabling direct canonicalization of the 3D mesh from the predicted depth maps. Here, instead of predicting high-dimensional skinning weights, we infer compressed skinning weights, i.e., 3-dimensional vector, with the aid of pre-trained MLP networks. We also introduce a forward skinning-based differentiable rendering scheme to merge the reconstructed results from multiple images. This scheme refines the initial mesh by reposing the canonical mesh via the forward skinning and by minimizing photometric and geometric errors between the rendered and the predicted results. Our optimization scheme considers the position and color of vertices as well as the joint angles for each image, thereby mitigating the negative effects of pose errors. We conduct extensive experiments to demonstrate the effectiveness of our method and compare our CanonicalFusion with state-of-the-art methods. Our source codes are available at https://github.com/jsshin98/CanonicalFusion.