Holoported Characters: Real-time Free-viewpoint Rendering of Humans from Sparse RGB Cameras

TL;DR

This paper introduces a real-time, high-resolution method for rendering realistic free-viewpoint videos of humans using only four sparse cameras, enabling immersive telepresence with detailed dynamic features.

Contribution

It presents a novel three-stage neural approach that achieves high-quality, real-time 4K rendering of humans from minimal camera views, handling complex clothing and expressions.

Findings

Real-time 4K rendering from four cameras

Handles detailed clothing wrinkles and facial expressions

Sets new benchmark for sparse-view human rendering

Abstract

We present the first approach to render highly realistic free-viewpoint videos of a human actor in general apparel, from sparse multi-view recording to display, in real-time at an unprecedented 4K resolution. At inference, our method only requires four camera views of the moving actor and the respective 3D skeletal pose. It handles actors in wide clothing, and reproduces even fine-scale dynamic detail, e.g. clothing wrinkles, face expressions, and hand gestures. At training time, our learning-based approach expects dense multi-view video and a rigged static surface scan of the actor. Our method comprises three main stages. Stage 1 is a skeleton-driven neural approach for high-quality capture of the detailed dynamic mesh geometry. Stage 2 is a novel solution to create a view-dependent texture using four test-time camera views as input. Finally, stage 3 comprises a new image-based refinement network rendering the final 4K image given the output from the previous stages. Our approach establishes a new benchmark for real-time rendering resolution and quality using sparse input camera views, unlocking possibilities for immersive telepresence.