Few-shot Neural Human Performance Rendering from Sparse RGBD Videos

TL;DR

This paper presents a novel few-shot neural human rendering method that produces photo-realistic free-view videos from sparse RGBD inputs by leveraging temporal and spatial redundancies, with a two-branch neural blending and adversarial training.

Contribution

It introduces a new approach for human rendering from sparse inputs, utilizing key-frame training, neural blending, and patch-based adversarial learning for detailed results.

Findings

Effective high-quality free-view rendering from sparse RGBD data.

Outperforms existing methods in challenging human performance scenarios.

Maintains detailed and realistic outputs with limited input data.

Abstract

Recent neural rendering approaches for human activities achieve remarkable view synthesis results, but still rely on dense input views or dense training with all the capture frames, leading to deployment difficulty and inefficient training overload. However, existing advances will be ill-posed if the input is both spatially and temporally sparse. To fill this gap, in this paper we propose a few-shot neural human rendering approach (FNHR) from only sparse RGBD inputs, which exploits the temporal and spatial redundancy to generate photo-realistic free-view output of human activities. Our FNHR is trained only on the key-frames which expand the motion manifold in the input sequences. We introduce a two-branch neural blending to combine the neural point render and classical graphics texturing pipeline, which integrates reliable observations over sparse key-frames. Furthermore, we adopt a patch-based adversarial training process to make use of the local redundancy and avoids over-fitting to the key-frames, which generates fine-detailed rendering results. Extensive experiments demonstrate the effectiveness of our approach to generate high-quality free view-point results for challenging human performances under the sparse setting.