NBAvatar: Neural Billboards Avatars with Realistic Hand-Face Interaction

TL;DR

NBAvatar is a novel neural rendering approach that creates realistic head avatars with accurate hand-face interaction, combining explicit primitives and neural rendering for high-quality, pose-consistent visualizations.

Contribution

It introduces a new avatar representation that effectively models non-rigid deformations from hand-face interactions using combined explicit and neural methods.

Findings

Achieves up to 30% LPIPS reduction in high-resolution rendering.

Outperforms existing methods in novel-view and novel-pose quality.

Improves PSNR, SSIM, and structural similarity over state-of-the-art approaches.

Abstract

We present NBAvatar - a method for realistic rendering of head avatars handling non-rigid deformations caused by hand-face interaction. We introduce a novel representation for animated avatars by combining the training of oriented planar primitives with neural rendering. Such a combination of explicit and implicit representations enables NBAvatar to handle temporally and pose-consistent geometry, along with fine-grained appearance details provided by the neural rendering technique. In our experiments, we demonstrate that NBAvatar implicitly learns color transformations caused by face-hand interactions and surpasses existing approaches in terms of novel-view and novel-pose rendering quality. Specifically, NBAvatar achieves up to 30% LPIPS reduction under high-resolution megapixel rendering compared to Gaussian-based avatar methods, while also improving PSNR and SSIM, and achieves higher structural similarity compared to the state-of-the-art hand-face interaction method InteractAvatar.