PaCo-FR: Patch-Pixel Aligned End-to-End Codebook Learning for Facial Representation Pre-training

TL;DR

PaCo-FR is an unsupervised facial representation pre-training method that combines structured masking, patch-based codebook, and spatial constraints to improve facial analysis tasks with limited labeled data.

Contribution

It introduces a novel framework integrating spatially coherent masking, a patch-based codebook, and spatial constraints for improved facial feature learning.

Findings

Achieves state-of-the-art results on facial analysis tasks.

Effective with only 2 million unlabeled images for pre-training.

Improves robustness to pose, occlusion, and lighting variations.

Abstract

Facial representation pre-training is crucial for tasks like facial recognition, expression analysis, and virtual reality. However, existing methods face three key challenges: (1) failing to capture distinct facial features and fine-grained semantics, (2) ignoring the spatial structure inherent to facial anatomy, and (3) inefficiently utilizing limited labeled data. To overcome these, we introduce PaCo-FR, an unsupervised framework that combines masked image modeling with patch-pixel alignment. Our approach integrates three innovative components: (1) a structured masking strategy that preserves spatial coherence by aligning with semantically meaningful facial regions, (2) a novel patch-based codebook that enhances feature discrimination with multiple candidate tokens, and (3) spatial consistency constraints that preserve geometric relationships between facial components. PaCo-FR achieves state-of-the-art performance across several facial analysis tasks with just 2 million unlabeled images for pre-training. Our method demonstrates significant improvements, particularly in scenarios with varying poses, occlusions, and lighting conditions. We believe this work advances facial representation learning and offers a scalable, efficient solution that reduces reliance on expensive annotated datasets, driving more effective facial analysis systems.