Collaborative Face Experts Fusion in Video Generation: Boosting Identity Consistency Across Large Face Poses

TL;DR

This paper introduces a novel fusion mechanism and a specialized dataset to improve identity preservation in face video generation across large poses, achieving superior results over existing methods.

Contribution

The paper proposes Collaborative Face Experts Fusion (CoFE) for better identity feature integration and a new large-scale pose-annotated dataset, LaFID-180K, for training and evaluation.

Findings

Outperforms state-of-the-art in face similarity and FID scores

Enhances identity consistency across large face poses

Improves semantic alignment with textual captions

Abstract

Current video generation models struggle with identity preservation under large face poses, primarily facing two challenges: the difficulty in exploring an effective mechanism to integrate identity features into DiT architectures, and the lack of targeted coverage of large face poses in existing open-source video datasets. To address these, we present two key innovations. First, we propose Collaborative Face Experts Fusion (CoFE), which dynamically fuses complementary signals from three specialized experts within the DiT backbone: an identity expert that captures cross-pose invariant features, a semantic expert that encodes high-level visual context, and a detail expert that preserves pixel-level attributes such as skin texture and color gradients. Second, we introduce a data curation pipeline comprising three key components: Face Constraints to ensure diverse large-pose coverage, Identity Consistency to maintain stable identity across frames, and Speech Disambiguation to align textual captions with actual speaking behavior. This pipeline yields LaFID-180K, a large-scale dataset of pose-annotated video clips designed for identity-preserving video generation. Experimental results on several benchmarks demonstrate that our approach significantly outperforms state-of-the-art methods in face similarity, FID, and CLIP semantic alignment. Project page: https://rain152.github.io/CoFE/.