SSAT: A Symmetric Semantic-Aware Transformer Network for Makeup Transfer and Removal

TL;DR

The paper introduces SSAT, a transformer network that achieves both makeup transfer and removal by learning semantic correspondences, resulting in more accurate and robust makeup editing across images and videos.

Contribution

A novel symmetric semantic-aware transformer network with a semantic correspondence module for simultaneous makeup transfer and removal.

Findings

Outperforms existing methods in visual accuracy

Effective in handling pose, expression, and occlusion variations

Successfully extended to video makeup transfer

Abstract

Makeup transfer is not only to extract the makeup style of the reference image, but also to render the makeup style to the semantic corresponding position of the target image. However, most existing methods focus on the former and ignore the latter, resulting in a failure to achieve desired results. To solve the above problems, we propose a unified Symmetric Semantic-Aware Transformer (SSAT) network, which incorporates semantic correspondence learning to realize makeup transfer and removal simultaneously. In SSAT, a novel Symmetric Semantic Corresponding Feature Transfer (SSCFT) module and a weakly supervised semantic loss are proposed to model and facilitate the establishment of accurate semantic correspondence. In the generation process, the extracted makeup features are spatially distorted by SSCFT to achieve semantic alignment with the target image, then the distorted makeup features are combined with unmodified makeup irrelevant features to produce the final result. Experiments show that our method obtains more visually accurate makeup transfer results, and user study in comparison with other state-of-the-art makeup transfer methods reflects the superiority of our method. Besides, we verify the robustness of the proposed method in the difference of expression and pose, object occlusion scenes, and extend it to video makeup transfer. Code will be available at https://gitee.com/sunzhaoyang0304/ssat-msp.