Evaluating the Efficacy of Skincare Product: A Realistic Short-Term Facial Pore Simulation

TL;DR

This paper introduces the first realistic simulation model for facial pore changes after skincare product use, enabling better skin health management and product evaluation through a two-step pipeline involving data collection and pore deformation modeling.

Contribution

The paper presents a novel simulation pipeline with a sliding window data cleaning method and a UNet-based pore segmentation, advancing facial skin simulation technology.

Findings

Realistic rendering of facial pore changes achieved

Effective data cleaning improves simulation accuracy

Pipeline facilitates future skincare research and product testing

Abstract

Simulating the effects of skincare products on face is a potential new way to communicate the efficacy of skincare products in skin diagnostics and product recommendations. Furthermore, such simulations enable one to anticipate his/her skin conditions and better manage skin health. However, there is a lack of effective simulations today. In this paper, we propose the first simulation model to reveal facial pore changes after using skincare products. Our simulation pipeline consists of 2 steps: training data establishment and facial pore simulation. To establish training data, we collect face images with various pore quality indexes from short-term (8-weeks) clinical studies. People often experience significant skin fluctuations (due to natural rhythms, external stressors, etc.,), which introduces large perturbations in clinical data. To address this problem, we propose a sliding window mechanism to clean data and select representative index(es) to represent facial pore changes. Facial pore simulation stage consists of 3 modules: UNet-based segmentation module to localize facial pores; regression module to predict time-dependent warping hyperparameters; and deformation module, taking warping hyperparameters and pore segmentation labels as inputs, to precisely deform pores accordingly. The proposed simulation is able to render realistic facial pore changes. And this work will pave the way for future research in facial skin simulation and skincare product developments.

Tables

Table 1. Table 1: Qualitative Evaluation in Face Pore Segmentation

Metrics	UNet-based	DoG-based
Dice	0.6663	0.4418
IoU	0.5139	0.2851
Precision	0.6790	0.4388
Accuracy	0.9936	0.9896

Table 2. Table 2: Random Forest Regression Analysis

Metrics	Time Window 1	Time Window 2	Time Window 3
$R^2$ Score	0.9905	0.9941	0.9564
MAE	0.6344±2.20	0.6039±1.85	1.3498±5.12

Table 3. Table 3: Quantitative Evaluation on Simulation Image Quality

Metrics	Simulation images
PIQE	11.86
NIQE	3.41

Equations

$$\text{L}=\text{Cross Entropy}+\text{F1 Score}.$$

$$f_s(r)=\left(1-{\left(\frac{r}{r_{\max }}-1\right)}^{2}a\right)r.$$

Full text

Introduction

Consumers prefer smooth and flawless skin that makes them look youthful and healthy. Skin texture plays a key role in the perception of human facial beauty [1, 2]. However, skin texture can appear rough and bumpy when facial pores enlarge. There are various exogenous and endogenous factors, such as gender, race, aging, and hormones that cause enlarged facial pores. Higher casual sebum levels in the nose and medial cheek area explain larger pores observed from these areas [3, 4]. Skincare products for pore care are widely available. Although there is a number of skincare diagnostic and recommendation capabilities available, to our knowledge, there is a lack of effective simulations today that reflect true facial pore transformation using available skincare products. Therefore, this paper proposes a complete pipeline to simulate the facial pore changes across a short-term period. Temporal analyses of changes in facial pores and simulation of these changes are helpful for the consumers to dynamically understand their skin and evaluate the potential benefit of skincare products. Such a realistic simulation not only guides customers to buy the right skincare products but also helps develop effective skincare technologies. Additionally, facial pore simulation can play a role, for example, in simulating the elongated effect of facial pores during the aging process. It enriches aging signs and contributes to the development of better aging models.

A realistic short-term simulation of the efficacy of skincare products builds upon truthful clinical studies. Similar to [5], clinical studies were conducted on 60 young Japanese females to reveal facial pore changes after applying specific skincare products A and B (in product code). The eMR Pro devices, designed for portable self-facial imaging with constant positioning and brightness [5], were provided for participants to capture side-face images three times a day (morning after wake up; morning after face wash; and evening after face wash). Facial pore changes correlate with the initial skin condition, which varies among the 60 participants. Following [6], various indexes are used to evaluate facial pore condition and we observe large perturbations in the index values due to daily skin fluctuations [5]. Based on this complex clinical data, we expect first to select valuable index(es) that reflect facial pore changes before and after using skincare products, regardless of different initial skin conditions. We further propose a sliding window mechanism to reduce the data perturbations efficiently and get the data ready for training.

Our next step is to perform a facial pore simulation. To realistically simulate changes in facial pores while maintaining image fidelity, we need to pay attention to three aspects: morphological modification of all facial pores, precise control of the modification to reflect the real changes, and the non-pore areas remaining unchanged. Nowadays, GAN-based architectures [7, 8, 9] are famous for generating high-quality images. However, these methods have limitations in maintaining non-pore facial region features. And the facial pore transformations after using skincare products correlate with the initial condition, which is also challenging for GAN-based methods to capture such changes. From Figure 1, we can see that GAN-based architecture is weak at simulating the detailed variation of facial pores (blurring effect in the cheek area) and maintaining the non-pore facial features.

To address the above-mentioned challenges, we propose a facial pore simulation model that consists of three modules: segmentation module, regression module, and deformation module. The segmentation module is to provide accurate spatial information by detecting and localizing visible and enlarged facial pores. Existing pore detection works [6, 10, 11, 12, 13, 14] take traditional approaches: setting threshold values carefully to segment facial pore features. Clinical imaging devices, such as, Visia-CR [6, 12] and Dermascore [11] are used to capture high-resolution images at the same time to control lighting conditions and posture. [10, 11, 12, 13, 14] are also limited to processing only small skin regions instead of full faces due to the threshold mechanism. By contrast, our UNet-based segmentation model is trained to learn robust facial pore features and to detect visible and enlarged pores in full side-face images. Our data-driven model detects pores with different sizes & shapes, and it can handle images with different lighting to a certain extent. In parallel, a random forest regression model is expected to learn the pattern of facial pore changes over time from the training data and to further predict the time-dependent warping hyperparameter. The warping hyperparameter is to control the degree of facial pore deformation. Lastly, in the deformation module, the local scaling warp method [33] is modified to calculate a flow-field grid with taking pore segmentation labels and warping hyperparameters as inputs. The flow-field grid assures that only facial pores are deformed, leaving the non-pore facial region unchanged. The original input image is then deformed accordingly to simulate facial pore changes after using specific skincare products.

To summarize, the contributions of this work are as follows:

• $\bullet$

We propose a suite of customized analytical tools to process complex facial pore clinical data. We demonstrate that the sliding window mechanism helps clean the data by reducing perturbations due to significant skin fluctuations.

• $\bullet$

We propose a facial pore simulation that consists of three modules: segmentation module, regression module, and deformation module. We show that by incorporating facial pore segmentation labels and predicted warping hyperparameters, the deformation module can precisely deform facial pores with accurate control, while leaving the rest of the face unchanged.

• $\bullet$

We propose a complete simulation pipeline that has two steps: training data establishment and facial pore simulation. Our results show that the simulation produces high-quality images which demonstrate realistic facial pore changes over time.

Related Work

Facial Pore detection

Facial pore detection is a challenging task as facial pores are delicate with different shapes (circular or elongated) [12, 18] & sizes (from 50 $\upmu$m to 500 $\upmu$m) [16]. It requires images to be sharp enough to contain pore-level features. Existing works [6, 11, 12, 13, 14, 15, 18, 19] require professional equipment to capture high-quality facial skin in a fixed pose under consistent light. [17, 19] only output the overall grading of facial pores. Traditional approaches are mainly used to detect facial pores: mark pores on melanin layers derived from digital images [16, 20]; using the difference of Gaussian (DoG) filters [6]; based on Fast fuzzy c-mean algorithm [14]. However, these methods are constrained to work on skin patches, and often perform poorly on images with varied lighting. Our proposed UNet-based model is capable of detecting skin pores on full side-face images with satisfactory performance and addresses different lighting problems.

Face Simulation

Existing GAN-based works [21, 22, 23, 24, 25, 26] focus on simulating the human face aging process. For instance, facial attributes are used to guide wavelet-based GANs [24] to simulate aging effect. [25] learns about people’s age progression by unraveling subject-specific features and age-specific effects. Triple-GAN [26] proposes to learn the interrelationships between different age groups. Aging simulation is at a larger scale than simulating skin changes. To the best of our knowledge, no work has been done to simulate the efficacy of skincare products on the face.

Methodology

Training Data Establishment

8-week clinical studies have been conducted on 60 young Japanese females from 22 to 34 years old: they used product A and/or B on the left/right side of their faces. Participants used eMR Pro devices [5] connected to smartphones to capture side-face images three times a day: morning after wake-up; morning after face wash; evening after face wash. Various pore statistics were measured using the method from [6]: pore count; total pore area; mean pore area; pore shape; orientation; mean L*/a*/b*-channel value in CIELAB color space. [5] examines the noticeable fluctuations in the skin that people experience every day, especially with facial pores. We use similar approaches from [5] to analyze these measured indexes while focusing on the 4-week observation period with applying products A and B. Our dataset consists of 12,531 images in total, and participants have different initial skin conditions with diverse pore size scales (from grade 0 to grade 5) [18]. To observe the facial pore changes among participants equally, we first normalize index value for every participant. On each observation day, a mean value of each index is computed to represent facial pore condition among the group. This averaging operation is designed to consider the different initial skin conditions of 60 participants. Linear regression analysis is then conducted to study index patterns before and after using skincare products. Valuable index(es) are to be selected to represent facial pore changes. Considering people have different degrees of skin fluctuation across days, we propose a $n$-day sliding window mechanism to reduce the data perturbations: gradually group $n$-day values in 1-day steps and then remove extreme values by the $n$-day mean and standard deviation. In addition, we observe that facial pore changes are developed slowly over the 4-week period of using skincare products, and we propose to split the 4-week observational duration into 3 discrete time windows to investigate skincare efficacy. The cleaned dataset contains 3,025 images and each participant has multiple sets of images to reflect their facial pore changes across time windows, which is used as the training set to develop a facial pore simulation model.

Facial Pore Simulation

UNet-based Facial Pore Segmentation

Pore segmentation labels lay the foundation for developing a good simulation of facial pore changes. In contrast with existing threshold-based traditional works [6, 10, 11, 12, 13, 14], we propose a deep learning-based data-driven model to detect visible and enlarged facial pores on side-face images captured by smartphones. We use 9,506 out of 12,531 images as the segmentation training set. It is time-consuming and labor-intensive if we were to manually annotate facial pores. Additionally, the simulation does not require exhaustive detection of facial pores. Hence, we propose a new approach to generate satisfactory labels to improve productivity. Inspired by [31], we use the Photocopy filter to extract facial details and then apply various post-processing operations to reduce excessive noise: morphological dilation and erosion; setting thresholds for pore area, size [16], and shape [12, 18]. Lastly, the pore segmentation labels were manually checked and the necessary refinements were made.

Witnessing the outstanding performance of UNet-based architectures in medical image segmentation, we train a pore segmentation model on 9,506 images using UNet [32] architecture. By measuring the similarity between the predicted segmentation label and the true segmentation label, the loss function penalizes the model to learn in the optimal direction. We add cross entropy loss and F1 score to form our loss function:

[TABLE]

Our selfie images are captured at 1920 x 1080 pixels. To avoid large memory overhead over training, images are cropped into patches with size of 256 x 256 with batch size of 24. RMSProp optimization algorithm is used with learning rate = 1e-5, weight decay = 1e-8, and momentum = 0.9.

Random Forest Regression

The regression module is to build up the relationship between the selected index and time window statistically. Our data indicate that participants experience different facial pore transformations with varied initial skin conditions, making it challenging to build an appropriate regression model. In addition, our data are considerably large in size. Therefore, we exploit Random Forest [28] as our regression model. Random Forest consists of a forest of classifying decision trees: it selects a subset of data randomly over training at each split and efficiently improves accuracy and controls over-fitting by using average values based on all aggregations [29, 30].

Facial Pore Deformation

Local scaling warp from [33] creates the possibility for precise manipulation. It achieves pixel-wisely manipulation by mapping points from input space to warping space without changing the colors, and the mapping is computed by:

[TABLE]

One minimum enclosing circle is found for each detected pore and the radius of each enclosing circle is $r_{max}$. Variable $r$ measures the distance between the target pixel and the center of the enclosing circle, and parameter $a$ controls the degree of deformation. Instead of implementing this method in an interactive manner, a well-fit random forest regression model is trained to predict parameter $a$. Based on Equation 2, mappings for all detected pores are gathered to compute a flow-field grid. Each size-2 vector in the flow-field grid is used to interpolate the corresponding output value. The bilinear interpolation method is used here and border values are used for out-of-bound grid locations. The face deformation operation is also optimized and it takes only about 5-10 seconds to manipulate one full side-face image of size 1920 x 1080.

Results

Our complete simulation pipeline consists of 2 stages: training data establishment and facial pore simulation. First, we show the analytical patterns of facial skin quality indexes over time. Next, we present the evaluation for facial pore segmentation, random regression, and facial pore simulation.

Training Data Establishment

Various indexes were measured in digital images. People’s skin fluctuates a lot within a day and/or from day to day, and we can observe large data perturbations in pore quality index values. After careful analysis of the statistics, we observe that index Pore_Area_total demonstrates a jittery decreasing pattern across 3 time windows (shown in the first column in Figure 3), and this pattern is consistent with the visual analysis by domain experts. Index Pore_Area_total is then selected as the representative index to reflect facial pore changes after skincare. With implementing the sliding window mechanism in the observation period, outliers are removed, and the data display fewer perturbations (shown in the second and third columns of Figure 3). The 3-day sliding window mechanism retains most data and it is then utilized to clean the data. After splitting the 4 observation week into 3 time windows, we further investigate the changes in value of index Pore_Area_total. The box plot also shows a jittery decreasing pattern in Figure 3, and it supports our observation.

Facial Pore Simulation

UNet-based Facial Pore Segmentation

A test set of 30 images is manually annotated for quantitative evaluation of facial pore segmentation. These test images are carefully selected from the cleaned facial pore simulation training set (consisting of 3,025 images), which considers different facial pore conditions and lighting conditions. As existing works are implemented in traditional threshold-based methods, we only compare our UNet-based pore segmentation model with the DoG-based method [6]. Multiple standard metric results in Table 1 show that the UNet-based segmentation model outperforms the DoG-based methods by a large margin. Figure 3 shows the visual comparison and we can conclude that our UNet-based segmentation model is capable of segmenting facial pores of different sizes and shapes. The model is also performing well in facial cheek area where we can observe specular effect in the first two input images in Figure 3.

Random Forest Regression

Random forest regression is trained to learn the relationship between metric Pore_Area_total and time windows. $R^{2}$ Score and Mean Average Error (MAE) are utilized here to evaluate regression performance. The results in Table 2 show the random forest regression model fits well with low error.

Facial Pore Deformation

As we have discussed previously, existing face simulation methods focus on age regression and their evaluation metrics are not applicable here. Similar to image super-resolution [34, 35], face image quality is a significant factor to develop a high-quality simulation system. In our dataset, side-face images are inevitably misaligned even with using the eMR Pro devices [5]. In addition, image registration methods do not work well for side-face images. Therefore, we use No-Reference Image Quality Assessment (NR-IQA) for evaluation: NIQE [36] and PIQE [37]. The detailed results in Table 3 indicate the high quality of the simulated images. In Figure 4, we show qualitative results of our model simulating the facial pore changes of 3 subjects after using skincare products. For each subject, the leftmost image shows the condition of his skin before using skin care products. Real images shown in the top row demonstrate his real skin condition after 10/20/30 days of skincare. Correspondingly, the bottom row presents the simulated skin condition of the facial pores for each time window (TW10, TW20, TW30). As can be seen, our model captures the delicate facial pore changes across the three time windows and modifies the input image to reflect realistically his facial pore changes over time. We can also observe that the model only modifies facial pore appearance while leaving other facial features unchanged, as shown in Figure 4.

Perception Study

We further conduct a perception study to evaluate our UNet-based segmentation model and facial pore simulation model. To be fair, we selected 40 images of facial pore segmentation results with diverse pore conditions under different light conditions. Similarly, 20 sets of simulation images were prepared.

We invited 30 respondents to this study: 15 women and 15 men, from ethnicity: Chinese and Indian. From the age range graph in Figure 7, we can see that more than 90% of the respondents are in their 20s, which is consistent with the young Japanese participants in our clinical studies. In addition, we asked the respondents to carefully self-assess their facial pore status with reference to a clinical standardized scale of pore size from [18]. The pore size scale pie chart in Figure 7 shows that the facial pore status is diverse among the 30 respondents, which shares a similar distribution to our clinical study.

Facial Pore Segmentation

Respondents were to ask 3 questions when observing 40 pore segmentation results:

• $\cdot$

Q1: How do you observe the overall detection results? (Choose one: Very Good; Good; Bad)

• $\cdot$

Q2: Are there any incorrectly detected pores? (Choose one: A few; Moderate; Many)

• $\cdot$

Q3: Overall, how does the algorithm perform? (Choose one: Very Good; Good; Fair; Poor; Very Poor)

The first and second questions are to evaluate the performance of the UNet-based pore segmentation model in terms of false negative and false positive respectively. Respondents answered Q1 and Q2 for each image and grade the overall performance in Q3 after viewing 40 images. As shown in Figure 7, 95% of the responses show satisfaction with our pore detection results (54% Very Good; 41% Good; and 5% Bad). In Q2, 60% of responses indicate there are a few false positives in segmentation results and 35% indicate a moderate number of false positives. Overall, more than 75% of the respondents feel good or very good at facial pore segmentation, with no negative comments.

Facial Pore Simulation

Our simulation progressively predicts facial pore changes at each time window. Respondents were invited to answer 3 questions by seeing 20 sets of images to evaluate each time window as well as the entire process.

• $\cdot$

Q1: Do the simulated images conform with the real images at TW10 (i.e., Time Window of 10-day using skincare)/TW20/TW30? (Choose one: Very Similar; Similar; Different)

• $\cdot$

Q2: Are you satisfied with the simulation of facial pore changes over time? (Choose one: Very Satisfied; Satisfied; Ambiguous; Unsatisfied; Very unsatisfied)

• $\cdot$

Q3: Overall, how does the simulation perform? (Choose one: Very Good; Good; Fair; Poor; Very Poor)

In Q1, respondents compared real and simulated images at each time window and scored their similarity in terms of facial pore condition only. And Q2 focused on whether the simulation model captured the pattern of facial pore changes across time. Respondents then graded the overall performance after viewing 20 sets of images in Q3. Figure 7 shows that more than 90% of the responses vote that the simulated images are similar or very similar to the real images at each time window. Over 80% of respondents rate the simulated facial pore changes as satisfied or very satisfied by looking at the 1-month duration. The bottom plot from Figure 7 shows that 90% of the respondents (27 persons) perceive that this simulation faithfully reflected the facial pore changes over 4 weeks of skincare use.

Conclusion

In this paper, we present a realistic short-term facial pore simulation model to evaluate the efficacy of skincare products. We demonstrate that the sliding window mechanism is useful in reducing data perturbations caused by daily fluctuations in skin. The complex facial pore clinical data is then reorganized and prepared for training. Moreover, to achieve a realistic simulation on facial pores while maintaining image fidelity (i.e., the non-pore facial areas remain unchanged), we propose a facial pore simulation that precisely localizes visible and enlarged facial pores and further modify facial pores morphologically to reflect facial pore changes after applying skincare product over a short-term. Our method delivers promising results for facial pore segmentation and facial pore simulation.

Acknowledgements

This work was carried out at the Rapid-Rich Object Search (ROSE) Lab, Nanyang Technological University (NTU), Singapore. The research is supported in part by ASTAR under it’s ASTAR MBRC Strategic Positioning Fund (SPF) – ASTAR-P&G Collaboration (Award APG2013/113) and in part by China-Singapore International Joint Research Institute under Grant 206-A018001. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of the ASTAR.