RIN: Textured Human Model Recovery and Imitation with a Single Image

TL;DR

RIN introduces a volume-based framework that reconstructs textured 3D human models from a single image, enabling effective imitation and view estimation without multi-view input, advancing 3D human modeling techniques.

Contribution

The paper presents a novel single-image 3D human reconstruction method using volume-based modeling and a U-Net-like translation network, improving texture recovery and pose estimation.

Findings

Effective textured 3D human model reconstruction from a single image.

Reliable back view estimation using the proposed network.

Competitive human imitation results with minimal input data.

Abstract

Human imitation has become topical recently, driven by GAN's ability to disentangle human pose and body content. However, the latest methods hardly focus on 3D information, and to avoid self-occlusion, a massive amount of input images are needed. In this paper, we propose RIN, a novel volume-based framework for reconstructing a textured 3D model from a single picture and imitating a subject with the generated model. Specifically, to estimate most of the human texture, we propose a U-Net-like front-to-back translation network. With both front and back images input, the textured volume recovery module allows us to color a volumetric human. A sequence of 3D poses then guides the colored volume via Flowable Disentangle Networks as a volume-to-volume translation task. To project volumes to a 2D plane during training, we design a differentiable depth-aware renderer. Our experiments demonstrate that our volume-based model is adequate for human imitation, and the back view can be estimated reliably using our network. While prior works based on either 2D pose or semantic map often fail for the unstable appearance of a human, our framework can still produce concrete results, which are competitive to those imagined from multi-view input.