X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent Attention

TL;DR

X-NeMo is a diffusion-based portrait animation method that effectively captures subtle facial expressions and prevents identity leakage by using a disentangled, latent motion descriptor controlled through cross-attention.

Contribution

It introduces a novel end-to-end training framework with a 1D identity-agnostic motion descriptor and dual GAN supervision, improving expressiveness and disentanglement in zero-shot facial reenactment.

Findings

Outperforms state-of-the-art methods in expression quality

Reduces identity leakage in facial reenactment

Captures fine-grained facial motions accurately

Abstract

We propose X-NeMo, a novel zero-shot diffusion-based portrait animation pipeline that animates a static portrait using facial movements from a driving video of a different individual. Our work first identifies the root causes of the key issues in prior approaches, such as identity leakage and difficulty in capturing subtle and extreme expressions. To address these challenges, we introduce a fully end-to-end training framework that distills a 1D identity-agnostic latent motion descriptor from driving image, effectively controlling motion through cross-attention during image generation. Our implicit motion descriptor captures expressive facial motion in fine detail, learned end-to-end from a diverse video dataset without reliance on pretrained motion detectors. We further enhance expressiveness and disentangle motion latents from identity cues by supervising their learning with a dual GAN decoder, alongside spatial and color augmentations. By embedding the driving motion into a 1D latent vector and controlling motion via cross-attention rather than additive spatial guidance, our design eliminates the transmission of spatial-aligned structural clues from the driving condition to the diffusion backbone, substantially mitigating identity leakage. Extensive experiments demonstrate that X-NeMo surpasses state-of-the-art baselines, producing highly expressive animations with superior identity resemblance. Our code and models are available for research.