LookinGood^{\pi}: Real-time Person-independent Neural Re-rendering for High-quality Human Performance Capture

TL;DR

LookinGood^{ extpi} is a real-time neural re-rendering method that enhances low-quality human performance capture results, generalizes well to unseen people, and produces high-fidelity images using a two-branch network guided by reconstructed geometry.

Contribution

The paper introduces a novel two-branch neural network that improves real-time human re-rendering quality and generalization to unseen individuals.

Findings

Outperforms state-of-the-art methods in image quality

Effective in generalizing to unseen people

Enhances detail fidelity in re-rendered images

Abstract

We propose LookinGood^{\pi}, a novel neural re-rendering approach that is aimed to (1) improve the rendering quality of the low-quality reconstructed results from human performance capture system in real-time; (2) improve the generalization ability of the neural rendering network on unseen people. Our key idea is to utilize the rendered image of reconstructed geometry as the guidance to assist the prediction of person-specific details from few reference images, thus enhancing the re-rendered result. In light of this, we design a two-branch network. A coarse branch is designed to fix some artifacts (i.e. holes, noise) and obtain a coarse version of the rendered input, while a detail branch is designed to predict "correct" details from the warped references. The guidance of the rendered image is realized by blending features from two branches effectively in the training of the detail branch, which improves both the warping accuracy and the details' fidelity. We demonstrate that our method outperforms state-of-the-art methods at producing high-fidelity images on unseen people.