MarioNETte: Few-shot Face Reenactment Preserving Identity of Unseen Targets

TL;DR

MarioNETte is a novel face reenactment framework that effectively preserves identity and handles unseen targets in few-shot scenarios by using attention, feature alignment, and landmark disentanglement.

Contribution

It introduces new components like the landmark transformer and target feature alignment to improve identity preservation in few-shot face reenactment.

Findings

Outperforms baseline methods in realism and identity preservation.

Effectively handles large pose variations and unseen identities.

Maintains high-quality reenactments despite target-driver mismatches.

Abstract

When there is a mismatch between the target identity and the driver identity, face reenactment suffers severe degradation in the quality of the result, especially in a few-shot setting. The identity preservation problem, where the model loses the detailed information of the target leading to a defective output, is the most common failure mode. The problem has several potential sources such as the identity of the driver leaking due to the identity mismatch, or dealing with unseen large poses. To overcome such problems, we introduce components that address the mentioned problem: image attention block, target feature alignment, and landmark transformer. Through attending and warping the relevant features, the proposed architecture, called MarioNETte, produces high-quality reenactments of unseen identities in a few-shot setting. In addition, the landmark transformer dramatically alleviates the identity preservation problem by isolating the expression geometry through landmark disentanglement. Comprehensive experiments are performed to verify that the proposed framework can generate highly realistic faces, outperforming all other baselines, even under a significant mismatch of facial characteristics between the target and the driver.