Real-Time Neural Character Rendering with Pose-Guided Multiplane Images

TL;DR

This paper introduces a real-time, pose-guided MPI synthesis method for photorealistic, animatable character rendering in real scenes, leveraging multi-view captures and depth-adaptive MPI for improved view synthesis and control.

Contribution

It presents a novel pose-guided MPI synthesis approach with depth-adaptive MPI, enabling real-time, photorealistic, and controllable character rendering from multi-view images.

Findings

Outperforms state-of-the-art methods in novel-view synthesis quality

Achieves real-time rendering of animated characters

Demonstrates robustness to camera registration errors

Abstract

We propose pose-guided multiplane image (MPI) synthesis which can render an animatable character in real scenes with photorealistic quality. We use a portable camera rig to capture the multi-view images along with the driving signal for the moving subject. Our method generalizes the image-to-image translation paradigm, which translates the human pose to a 3D scene representation -- MPIs that can be rendered in free viewpoints, using the multi-views captures as supervision. To fully cultivate the potential of MPI, we propose depth-adaptive MPI which can be learned using variable exposure images while being robust to inaccurate camera registration. Our method demonstrates advantageous novel-view synthesis quality over the state-of-the-art approaches for characters with challenging motions. Moreover, the proposed method is generalizable to novel combinations of training poses and can be explicitly controlled. Our method achieves such expressive and animatable character rendering all in real time, serving as a promising solution for practical applications.