Learning Motion-Dependent Appearance for High-Fidelity Rendering of Dynamic Humans from a Single Camera

TL;DR

This paper introduces a novel method for high-fidelity rendering of dynamic humans from a single camera by modeling motion-dependent appearance using an equivariant representation that captures complex cloth deformations.

Contribution

It proposes a compact, motion-conditioned appearance model with an equivariant encoder and a multi-task decoder for realistic dynamic human rendering from minimal observations.

Findings

Generates temporally coherent videos of dynamic humans from a single view.

Achieves high-fidelity rendering for unseen poses and views.

Outperforms existing methods in realism and temporal consistency.

Abstract

Appearance of dressed humans undergoes a complex geometric transformation induced not only by the static pose but also by its dynamics, i.e., there exists a number of cloth geometric configurations given a pose depending on the way it has moved. Such appearance modeling conditioned on motion has been largely neglected in existing human rendering methods, resulting in rendering of physically implausible motion. A key challenge of learning the dynamics of the appearance lies in the requirement of a prohibitively large amount of observations. In this paper, we present a compact motion representation by enforcing equivariance -- a representation is expected to be transformed in the way that the pose is transformed. We model an equivariant encoder that can generate the generalizable representation from the spatial and temporal derivatives of the 3D body surface. This learned representation is decoded by a compositional multi-task decoder that renders high fidelity time-varying appearance. Our experiments show that our method can generate a temporally coherent video of dynamic humans for unseen body poses and novel views given a single view video.