DermSynth3D: Synthesis of in-the-wild Annotated Dermatology Images

TL;DR

DermSynth3D is a novel framework that synthesizes realistic dermatology images with detailed annotations from 3D models, addressing data scarcity and diversity issues in dermatological deep learning applications.

Contribution

It introduces a differentiable rendering-based method to generate diverse, annotated 2D dermatology images from 3D textured meshes, improving dataset quality for deep learning.

Findings

Synthetic data improves model training for dermatology tasks.

Models trained on synthetic data perform well on real images.

The framework enables customizable dataset creation.

Abstract

In recent years, deep learning (DL) has shown great potential in the field of dermatological image analysis. However, existing datasets in this domain have significant limitations, including a small number of image samples, limited disease conditions, insufficient annotations, and non-standardized image acquisitions. To address these shortcomings, we propose a novel framework called DermSynth3D. DermSynth3D blends skin disease patterns onto 3D textured meshes of human subjects using a differentiable renderer and generates 2D images from various camera viewpoints under chosen lighting conditions in diverse background scenes. Our method adheres to top-down rules that constrain the blending and rendering process to create 2D images with skin conditions that mimic in-the-wild acquisitions, ensuring more meaningful results. The framework generates photo-realistic 2D dermoscopy images and the corresponding dense annotations for semantic segmentation of the skin, skin conditions, body parts, bounding boxes around lesions, depth maps, and other 3D scene parameters, such as camera position and lighting conditions. DermSynth3D allows for the creation of custom datasets for various dermatology tasks. We demonstrate the effectiveness of data generated using DermSynth3D by training DL models on synthetic data and evaluating them on various dermatology tasks using real 2D dermatological images. We make our code publicly available at https://github.com/sfu-mial/DermSynth3D.