What to Preserve and What to Transfer: Faithful, Identity-Preserving Diffusion-based Hairstyle Transfer

TL;DR

This paper introduces HairFusion, a diffusion-based hairstyle transfer model that effectively handles real-world scenarios by accurately aligning and blending hairstyles while preserving facial features, overcoming limitations of prior StyleGAN-based methods.

Contribution

The paper proposes a novel one-stage diffusion model with hair-agnostic representation and cross-attention alignment for robust hairstyle transfer in challenging conditions.

Findings

Achieves state-of-the-art results in hairstyle transfer quality.

Effectively preserves facial features and background during transfer.

Handles extreme head pose variations better than previous methods.

Abstract

Hairstyle transfer is a challenging task in the image editing field that modifies the hairstyle of a given face image while preserving its other appearance and background features. The existing hairstyle transfer approaches heavily rely on StyleGAN, which is pre-trained on cropped and aligned face images. Hence, they struggle to generalize under challenging conditions such as extreme variations of head poses or focal lengths. To address this issue, we propose a one-stage hairstyle transfer diffusion model, HairFusion, that applies to real-world scenarios. Specifically, we carefully design a hair-agnostic representation as the input of the model, where the original hair information is thoroughly eliminated. Next, we introduce a hair align cross-attention (Align-CA) to accurately align the reference hairstyle with the face image while considering the difference in their head poses. To enhance the preservation of the face image's original features, we leverage adaptive hair blending during the inference, where the output's hair regions are estimated by the cross-attention map in Align-CA and blended with non-hair areas of the face image. Our experimental results show that our method achieves state-of-the-art performance compared to the existing methods in preserving the integrity of both the transferred hairstyle and the surrounding features. The codes are available at https://github.com/cychungg/HairFusion