A Computer Vision Application for Assessing Facial Acne Severity from Selfie Images

TL;DR

This paper presents a novel deep learning model that accurately assesses facial acne severity from selfies, outperforming many dermatologists, and is deployed as a mobile app for patient use.

Contribution

The study introduces a new image rolling data augmentation technique and demonstrates the first deep learning-based acne severity assessment from selfies.

Findings

Model outperforms over half of dermatologists on test images.

Introduces a novel image augmentation method to improve CNN generalization.

First deep learning solution for acne assessment using selfie images.

Abstract

We worked with Nestle SHIELD (Skin Health, Innovation, Education, and Longevity Development, NSH) to develop a deep learning model that is able to assess acne severity from selfie images as accurate as dermatologists. The model was deployed as a mobile application, providing patients an easy way to assess and track the progress of their acne treatment. NSH acquired 4,700 selfie images for this study and recruited 11 internal dermatologists to label them in five categories: 1-Clear, 2- Almost Clear, 3-Mild, 4-Moderate, 5-Severe. Using OpenCV to detect facial landmarks we cut specific skin patches from the selfie images in order to minimize irrelevant background. We then applied a transfer learning approach by extracting features from the patches using a ResNet 152 pre-trained model, followed by a fully connected layer trained to approximate the desired severity rating. To address the problem of spatial sensitivity of CNN models, we introduce a new image rolling data augmentation approach, effectively causing acne lesions appeared in more locations in the training images. Our results demonstrate that this approach improved the generalization of the CNN model, outperforming more than half of the panel of human dermatologists on test images. To our knowledge, this is the first deep learning-based solution for acne assessment using selfie images.