LiveHand: Real-time and Photorealistic Neural Hand Rendering

TL;DR

LiveHand introduces a neural-implicit method for real-time, photorealistic rendering of human hands, capturing detailed appearance and pose-dependent effects, advancing applications in extended reality and gaming.

Contribution

It is the first neural-implicit approach enabling real-time, photorealistic hand rendering with pose-dependent effects, combining neural radiance fields, super-resolution, and mesh guidance.

Findings

Achieves real-time, photorealistic hand rendering.

Effectively models pose- and view-dependent appearance.

Demonstrates a live demo with high-quality results.

Abstract

The human hand is the main medium through which we interact with our surroundings, making its digitization an important problem. While there are several works modeling the geometry of hands, little attention has been paid to capturing photo-realistic appearance. Moreover, for applications in extended reality and gaming, real-time rendering is critical. We present the first neural-implicit approach to photo-realistically render hands in real-time. This is a challenging problem as hands are textured and undergo strong articulations with pose-dependent effects. However, we show that this aim is achievable through our carefully designed method. This includes training on a low-resolution rendering of a neural radiance field, together with a 3D-consistent super-resolution module and mesh-guided sampling and space canonicalization. We demonstrate a novel application of perceptual loss on the image space, which is critical for learning details accurately. We also show a live demo where we photo-realistically render the human hand in real-time for the first time, while also modeling pose- and view-dependent appearance effects. We ablate all our design choices and show that they optimize for rendering speed and quality. Video results and our code can be accessed from https://vcai.mpi-inf.mpg.de/projects/LiveHand/