SHMT: Self-supervised Hierarchical Makeup Transfer via Latent Diffusion Models

TL;DR

This paper introduces SHMT, a self-supervised hierarchical makeup transfer method using latent diffusion models that improves makeup fidelity and style diversity without relying on paired training data.

Contribution

The paper proposes a novel self-supervised hierarchical makeup transfer approach with a Laplacian pyramid decomposition and an iterative dual alignment module, addressing data guidance and style diversity challenges.

Findings

Outperforms existing methods in makeup transfer fidelity.

Effectively handles diverse makeup styles.

Demonstrates superior qualitative and quantitative results.

Abstract

This paper studies the challenging task of makeup transfer, which aims to apply diverse makeup styles precisely and naturally to a given facial image. Due to the absence of paired data, current methods typically synthesize sub-optimal pseudo ground truths to guide the model training, resulting in low makeup fidelity. Additionally, different makeup styles generally have varying effects on the person face, but existing methods struggle to deal with this diversity. To address these issues, we propose a novel Self-supervised Hierarchical Makeup Transfer (SHMT) method via latent diffusion models. Following a "decoupling-and-reconstruction" paradigm, SHMT works in a self-supervised manner, freeing itself from the misguidance of imprecise pseudo-paired data. Furthermore, to accommodate a variety of makeup styles, hierarchical texture details are decomposed via a Laplacian pyramid and selectively introduced to the content representation. Finally, we design a novel Iterative Dual Alignment (IDA) module that dynamically adjusts the injection condition of the diffusion model, allowing the alignment errors caused by the domain gap between content and makeup representations to be corrected. Extensive quantitative and qualitative analyses demonstrate the effectiveness of our method. Our code is available at \url{https://github.com/Snowfallingplum/SHMT}.