MonoHuman: Animatable Human Neural Field from Monocular Video

TL;DR

MonoHuman introduces a novel neural framework that enables view-consistent, high-fidelity human avatar rendering from monocular videos, effectively modeling complex motions and generalizing to unseen poses.

Contribution

The paper presents a new deformation modeling approach with bidirectional constraints and keyframe correspondence reasoning for realistic, view-consistent avatar animation from monocular videos.

Findings

Outperforms state-of-the-art methods in view consistency and fidelity.

Successfully generalizes to unseen human poses.

Produces high-quality, multi-view consistent avatar renderings.

Abstract

Animating virtual avatars with free-view control is crucial for various applications like virtual reality and digital entertainment. Previous studies have attempted to utilize the representation power of the neural radiance field (NeRF) to reconstruct the human body from monocular videos. Recent works propose to graft a deformation network into the NeRF to further model the dynamics of the human neural field for animating vivid human motions. However, such pipelines either rely on pose-dependent representations or fall short of motion coherency due to frame-independent optimization, making it difficult to generalize to unseen pose sequences realistically. In this paper, we propose a novel framework MonoHuman, which robustly renders view-consistent and high-fidelity avatars under arbitrary novel poses. Our key insight is to model the deformation field with bi-directional constraints and explicitly leverage the off-the-peg keyframe information to reason the feature correlations for coherent results. Specifically, we first propose a Shared Bidirectional Deformation module, which creates a pose-independent generalizable deformation field by disentangling backward and forward deformation correspondences into shared skeletal motion weight and separate non-rigid motions. Then, we devise a Forward Correspondence Search module, which queries the correspondence feature of keyframes to guide the rendering network. The rendered results are thus multi-view consistent with high fidelity, even under challenging novel pose settings. Extensive experiments demonstrate the superiority of our proposed MonoHuman over state-of-the-art methods.