RelightableHands: Efficient Neural Relighting of Articulated Hand Models

TL;DR

This paper introduces a neural relighting method for animated, personalized hand models that achieves real-time, photorealistic rendering under novel lighting conditions by leveraging physics-inspired features for better generalization.

Contribution

It proposes a novel teacher-student framework with physics-inspired features for efficient, real-time neural relighting of articulated hands, improving generalization and rendering quality.

Findings

Outperforms baseline approaches in relighting quality.

Achieves real-time photorealistic hand rendering.

Demonstrates effective generalization to unseen poses and illuminations.

Abstract

We present the first neural relighting approach for rendering high-fidelity personalized hands that can be animated in real-time under novel illumination. Our approach adopts a teacher-student framework, where the teacher learns appearance under a single point light from images captured in a light-stage, allowing us to synthesize hands in arbitrary illuminations but with heavy compute. Using images rendered by the teacher model as training data, an efficient student model directly predicts appearance under natural illuminations in real-time. To achieve generalization, we condition the student model with physics-inspired illumination features such as visibility, diffuse shading, and specular reflections computed on a coarse proxy geometry, maintaining a small computational overhead. Our key insight is that these features have strong correlation with subsequent global light transport effects, which proves sufficient as conditioning data for the neural relighting network. Moreover, in contrast to bottleneck illumination conditioning, these features are spatially aligned based on underlying geometry, leading to better generalization to unseen illuminations and poses. In our experiments, we demonstrate the efficacy of our illumination feature representations, outperforming baseline approaches. We also show that our approach can photorealistically relight two interacting hands at real-time speeds. https://sh8.io/#/relightable_hands