Neural Re-Rendering of Humans from a Single Image

TL;DR

This paper introduces a neural re-rendering method that generates realistic images of humans in new poses and viewpoints from a single input image, using a parametric body model and learned appearance features.

Contribution

The novel approach combines a parametric mesh with a learned high-dimensional UV feature map for detailed appearance encoding, improving re-rendering quality over previous methods.

Findings

Produces higher quality re-rendered images than existing methods.

Effectively captures appearance variations across poses and clothing.

End-to-end neural network architecture for pose and viewpoint changes.

Abstract

Human re-rendering from a single image is a starkly under-constrained problem, and state-of-the-art algorithms often exhibit undesired artefacts, such as over-smoothing, unrealistic distortions of the body parts and garments, or implausible changes of the texture. To address these challenges, we propose a new method for neural re-rendering of a human under a novel user-defined pose and viewpoint, given one input image. Our algorithm represents body pose and shape as a parametric mesh which can be reconstructed from a single image and easily reposed. Instead of a colour-based UV texture map, our approach further employs a learned high-dimensional UV feature map to encode appearance. This rich implicit representation captures detailed appearance variation across poses, viewpoints, person identities and clothing styles better than learned colour texture maps. The body model with the rendered feature maps is fed through a neural image-translation network that creates the final rendered colour image. The above components are combined in an end-to-end-trained neural network architecture that takes as input a source person image, and images of the parametric body model in the source pose and desired target pose. Experimental evaluation demonstrates that our approach produces higher quality single image re-rendering results than existing methods.