High-Fidelity Relightable Monocular Portrait Animation with Lighting-Controllable Video Diffusion Model

TL;DR

This paper introduces LCVD, a novel diffusion model that enables high-fidelity, relightable portrait animation by separating intrinsic and extrinsic features for nuanced control over lighting, pose, and expression.

Contribution

It proposes a feature space separation within a diffusion model to achieve relightable portrait animation, addressing limitations of previous methods.

Findings

Outperforms state-of-the-art in lighting realism

Achieves higher image quality and video consistency

Sets a new benchmark in relightable portrait animation

Abstract

Relightable portrait animation aims to animate a static reference portrait to match the head movements and expressions of a driving video while adapting to user-specified or reference lighting conditions. Existing portrait animation methods fail to achieve relightable portraits because they do not separate and manipulate intrinsic (identity and appearance) and extrinsic (pose and lighting) features. In this paper, we present a Lighting Controllable Video Diffusion model (LCVD) for high-fidelity, relightable portrait animation. We address this limitation by distinguishing these feature types through dedicated subspaces within the feature space of a pre-trained image-to-video diffusion model. Specifically, we employ the 3D mesh, pose, and lighting-rendered shading hints of the portrait to represent the extrinsic attributes, while the reference represents the intrinsic attributes. In the training phase, we employ a reference adapter to map the reference into the intrinsic feature subspace and a shading adapter to map the shading hints into the extrinsic feature subspace. By merging features from these subspaces, the model achieves nuanced control over lighting, pose, and expression in generated animations. Extensive evaluations show that LCVD outperforms state-of-the-art methods in lighting realism, image quality, and video consistency, setting a new benchmark in relightable portrait animation.