Real-time Free-view Human Rendering from Sparse-view RGB Videos using Double Unprojected Textures

TL;DR

This paper introduces Double Unprojected Textures, a novel method for real-time, photorealistic human rendering from sparse RGB views, disentangling geometry and appearance for improved quality and robustness.

Contribution

It proposes a new approach that separates coarse geometry estimation from appearance synthesis, enhancing visual quality and robustness in real-time rendering from sparse views.

Findings

Achieves 4K photorealistic rendering in real-time.

Outperforms state-of-the-art methods quantitatively and qualitatively.

Reduces artifacts and improves alignment with input views.

Abstract

Real-time free-view human rendering from sparse-view RGB inputs is a challenging task due to the sensor scarcity and the tight time budget. To ensure efficiency, recent methods leverage 2D CNNs operating in texture space to learn rendering primitives. However, they either jointly learn geometry and appearance, or completely ignore sparse image information for geometry estimation, significantly harming visual quality and robustness to unseen body poses. To address these issues, we present Double Unprojected Textures, which at the core disentangles coarse geometric deformation estimation from appearance synthesis, enabling robust and photorealistic 4K rendering in real-time. Specifically, we first introduce a novel image-conditioned template deformation network, which estimates the coarse deformation of the human template from a first unprojected texture. This updated geometry is then used to apply a second and more accurate texture unprojection. The resulting texture map has fewer artifacts and better alignment with input views, which benefits our learning of finer-level geometry and appearance represented by Gaussian splats. We validate the effectiveness and efficiency of the proposed method in quantitative and qualitative experiments, which significantly surpasses other state-of-the-art methods. Project page: https://vcai.mpi-inf.mpg.de/projects/DUT/