# Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition

## TL;DR

This paper introduces a novel method for extracting makeup from a single 2D portrait to generate 3D facial models, enabling applications like makeup transfer and editing with improved accuracy and robustness.

## Contribution

It is the first approach to extract 3D makeup from 2D images using inverse rendering and inpainting, creating a large-scale dataset and parametric model for 3D faces.

## Key findings

- Effective extraction of makeup aligned in UV space.
- Robust makeup transfer and illumination-aware editing.
- Creation of a large-scale makeup dataset and parametric model.

## Abstract

Facial makeup enriches the beauty of not only real humans but also virtual characters; therefore, makeup for 3D facial models is highly in demand in productions. However, painting directly on 3D faces and capturing real-world makeup are costly, and extracting makeup from 2D images often struggles with shading effects and occlusions. This paper presents the first method for extracting makeup for 3D facial models from a single makeup portrait. Our method consists of the following three steps. First, we exploit the strong prior of 3D morphable models via regression-based inverse rendering to extract coarse materials such as geometry and diffuse/specular albedos that are represented in the UV space. Second, we refine the coarse materials, which may have missing pixels due to occlusions. We apply inpainting and optimization. Finally, we extract the bare skin, makeup, and an alpha matte from the diffuse albedo. Our method offers various applications for not only 3D facial models but also 2D portrait images. The extracted makeup is well-aligned in the UV space, from which we build a large-scale makeup dataset and a parametric makeup model for 3D faces. Our disentangled materials also yield robust makeup transfer and illumination-aware makeup interpolation/removal without a reference image.