Semi-parametric Makeup Transfer via Semantic-aware Correspondence

TL;DR

This paper introduces a semi-parametric makeup transfer method that combines non-parametric semantic-aware correspondence with parametric decoding to effectively handle pose, expression, and occlusion differences, improving visual quality and robustness.

Contribution

It proposes a novel semi-parametric framework with a semantic-aware correspondence module for more accurate makeup transfer.

Findings

Outperforms existing methods in visual quality and robustness.

Effectively handles pose, expression, and occlusion discrepancies.

Demonstrates flexibility and high-quality results in experiments.

Abstract

The large discrepancy between the source non-makeup image and the reference makeup image is one of the key challenges in makeup transfer. Conventional approaches for makeup transfer either learn disentangled representation or perform pixel-wise correspondence in a parametric way between two images. We argue that non-parametric techniques have a high potential for addressing the pose, expression, and occlusion discrepancies. To this end, this paper proposes a \textbf{S}emi-\textbf{p}arametric \textbf{M}akeup \textbf{T}ransfer (SpMT) method, which combines the reciprocal strengths of non-parametric and parametric mechanisms. The non-parametric component is a novel \textbf{S}emantic-\textbf{a}ware \textbf{C}orrespondence (SaC) module that explicitly reconstructs content representation with makeup representation under the strong constraint of component semantics. The reconstructed representation is desired to preserve the spatial and identity information of the source image while "wearing" the makeup of the reference image. The output image is synthesized via a parametric decoder that draws on the reconstructed representation. Extensive experiments demonstrate the superiority of our method in terms of visual quality, robustness, and flexibility. Code and pre-trained model are available at \url{https://github.com/AnonymScholar/SpMT.