Early-Stage Melanoma Benchmark Dataset

TL;DR

A new dataset called EMB helps improve early melanoma detection by providing labeled images for deep learning models.

Contribution

The EMB dataset introduces labeled images for early-stage melanoma detection, enabling T-category-specific analysis and cross-dataset benchmarking.

Findings

State-of-the-art models show reduced performance on EMB compared to ISIC datasets, especially for in situ and thin melanomas.

EMB enables T-category-specific analysis and cross-dataset benchmarking for early-stage melanoma detection.

The dataset highlights the challenges in detecting early-stage melanomas using current deep learning models.

Abstract

The Early-Stage Melanoma Benchmark (EMB) dataset was designed to support the development and evaluation of deep learning models for early melanoma detection. Existing datasets often lack information on melanoma stage or Breslow thickness, limiting researchers’ ability to accurately assess models in the early-stage melanoma detection task. EMB addresses this gap by providing over 1100 dermoscopic and clinical melanoma images labeled according to T-category in TNM classification. The dataset is curated from public sources and filtered to avoid overlap with ISIC training data. Several state-of-the-art models were evaluated on EMB, revealing a significant performance drop, particularly for in situ and thin melanomas, compared to results on standard ISIC datasets. This highlights the challenges of early-stage detection and the need for dedicated datasets. EMB enables cross-dataset benchmarking and T-category-specific analysis, offering a valuable resource for assessing clinical applicability and the robustness of automated melanoma diagnosis systems.

Background: The early detection of melanoma is crucial for improving patient outcomes, as survival rates decline dramatically with disease progression. Despite significant achievements in deep learning methods for skin lesion analysis, several challenges limit their effectiveness in clinical practice. One of the key issues is the lack of knowledge about the melanoma stage distribution in the training data, raising concerns about the ability of these models to detect early-stage melanoma accurately. Additionally, publicly available datasets that include detailed information on melanoma stage and tumor thickness remain scarce, restricting researchers from developing and benchmarking methods specifically tailored for early diagnosis. Another major limitation is the lack of cross-dataset evaluations. Most deep learning models are tested on the same dataset they were trained on, so they fail to assess their generalization ability when applied to unseen data. This reduces their reliability in real-world clinical settings. Methods: We introduce an early-stage melanoma benchmark dataset to address these issues, featuring images labeled according to T-category based on Breslow thickness. Results: We evaluated several state-of-the-art deep learning models on this dataset and observed a significant drop in performance compared to their results on the ISIC Challenge datasets. Conclusions: This finding highlights the models’ limited capability in detecting early-stage melanoma. This work seeks to advance the development and clinical applicability of automated melanoma diagnostic systems by providing a resource for T-category-specific analysis and supporting cross-dataset evaluation.

Figures

Full text

1. Introduction

Melanoma is one of the deadliest forms of skin cancer, but early detection significantly improves the chances of successful treatment. According to the U.S. National Cancer Institute, the 5-year survival rate for patients diagnosed at localized stages (0, I, II) is as high as 97.6%. In contrast, survival rates decrease sharply to 60.3% when diagnosed at a regional stage (III) and drop further to 16.2% at the metastatic stage (IV) [1]. These statistics highlight the importance of melanoma diagnosis in the early stages.

Recent advances in deep learning have accelerated the development of automated methods for skin lesion diagnosis, leading to a growing number of AI models designed specifically for dermatology [2,3,4,5]. The availability of relatively large, publicly accessible skin lesion datasets has further propelled progress in this field, enabling the refinement of deep-learning-based diagnostic approaches. The adoption of such automated systems for melanoma detection holds significant potential for improving early diagnosis and, consequently, enhancing patient survival rates.

Numerous studies have explored early melanoma detection using dermoscopic images with deep learning techniques [6]. For instance, Alenezi et al. [7] used a lesion segmentation mask to extract the area of interest for subsequent classification to improve melanoma diagnosis. Lin et al. [4] used YOLOv5 for the detection of different melanoma types on RGB images converted into hyperspectral images. A framework for explainable melanoma diagnosis that integrates features extracted from the lesion, population information, and patient metadata was developed by Akash et al. [8]. This framework uses a transformer architecture that allows for the leveraging of evidence based on global context derived from all patient lesions and associated metadata. Other notable contributions were made by Elshahawy et al. [5], who used YOLOv5 for melanoma detection on dermoscopic images in real time. Mahmud et al. [9] proposed an Xception-based method for early-stage melanoma detection. They also used extensive preprocessing techniques, including artifact removal and hair removal, along with data augmentation, to improve model generalization. While these described methods focus on early-stage melanoma detection, a critical gap persists, i.e., they frequently lack an analysis of model performance that explicitly verifies their efficiency on early-stage melanomas, often relying on broader performance metrics that may not fully capture this specific capability. Yu et al. [10] addressed the important issue of neglecting temporal dependencies in melanoma progression. Their approach leverages changes in the lesion over time to enhance early melanoma detection. This is accomplished by aggregating information from temporally aligned sequential images through a spatio-temporal network. Additionally, they implemented knowledge distillation to transfer insights from later time points to earlier ones by penalizing inconsistent predictions across different time steps.

Several dermatological AI methods have also been proposed for classifying Breslow thickness into intervals. For example, Polesie et al. [11] show that clinicians outperform a de novo convolutional neural network in classification between in situ and invasive melanoma, but pre-trained ResNet-50 achieved similar performance to the human readers. It is a challenging task due to the limited availability of data. To mitigate the scarcity of Breslow thickness-labeled images, knowledge distillation was employed to enhance classification performance in two tasks: distinguishing in situ from invasive melanoma, and categorizing melanoma into in situ, thin, and thick classes [12]. A different approach was proposed by Jaworek-Korjakowska et al. [13], who used the SMOTE oversampling method and the VGG-19 model to classify Breslow thickness into three intervals. Nogales et al. [14] used a pretrained ConvNeXt and focal loss for the classification of melanoma into two classes based on Breslow depth. Described studies rely either on private or difficult-to-access datasets [11,13] or on Breslow thickness labels that do not correspond to the T-category definitions of the AJCC staging system [15].

Despite recent progress, existing lesion analysis methods face several limitations. First, the majority of datasets suffer from a lack of detailed metadata from pathology reports such as Breslow thickness, presence of ulceration, regression, dermal mitotic rate, tumor-infiltrating lymphocytes, preexisting nevus, satelitosis, and perivascular and perineural invasion, making it difficult to assess how well these methods perform, especially on early-stage lesions. Specifically, Breslow thickness and ulceration determine the T-category in the TNM classification used for disease staging. Without this information, the distribution of early/advanced melanomas remains unknown. Since early detection of melanoma is a primary objective of deep learning models, the lack of dedicated datasets makes it difficult to assess their effectiveness accurately. This gap poses a significant challenge to the development of reliable diagnostic tools, as early-stage melanoma can exhibit visual similarities to benign nevi, borderline lesions, melanosis, or even scars [16]. These variations in appearance across stages could pose a challenge for deep learning models, potentially affecting classification performance. Additionally, early-stage melanoma often exhibits subtle features, making it more challenging to differentiate from benign lesions and complicating accurate diagnosis. As a result, an imbalanced distribution of melanoma stages in datasets can introduce bias into deep learning models. When trained on data with unknown stage distribution, models may not effectively address early detection, presumably failing to identify early-stage melanoma accurately.

Secondly, the wide availability of large open-access datasets presents a challenge, as nearly all research projects rely on the same training data, while cross-dataset evaluation remains vastly underutilized. For example, only 36 of 176 skin lesion segmentation methods studied in [17] used cross-dataset evaluation to test their models. Moreover, only 23.6% of works studied in [18] used an external dataset for evaluation. Without testing across diverse datasets, it is challenging to determine whether a method will perform robustly in real-world clinical settings.

Assessing the performance of deep learning models is a critical medical and legal concern, prompting the curation of specialized datasets for this purpose. Researchers can use general benchmarks to evaluate the robustness of deep learning models [19]. Additionally, specialized skin lesion analysis datasets exist, such as skin tone diversity benchmarks for lesion diagnosis [20,21] and datasets that categorize images by Breslow thickness bins rather than providing exact values [15]. While there are multiple datasets containing skin lesion images, it is not a common practice that models are tested with multiple datasets from different sources, and no further analysis of the bias and robustness of the models is provided [18].

We propose an early-stage melanoma benchmark dataset (EMB) consisting of melanoma images with Breslow thickness and T-category. Data was collected from known public sources and delivered in a standardized format for future use. We also present a detailed evaluation of multiple models on the curated dataset. Our results confirm that models perform worse on a dataset composed of thin melanoma images compared to on a dataset with an unknown stage distribution. The EMB dataset addresses the gaps described above as it provides labeled images with an emphasis on early-stage melanoma and is designed to facilitate cross-dataset benchmarking, offering a new standard for assessing the reliability and accuracy of automated melanoma diagnostic tools. The dataset and code used to prepare the data are available at GitHub https://github.com/Oichii/EMB (accessed on 20 June 2025).

2. Methods

The early-stage melanoma benchmark dataset was developed to support research on automated melanoma diagnosis with a particular focus on early detection. It consists of melanoma images paired with corresponding thickness information provided in millimeters. The thickness data was extracted from image descriptions or metadata found in archival sources. Each lesion was categorized into a T-category label based on its thickness.

2.1. Data Collection

Images for the datasets came from two publicly available repositories: the ISIC Archive [22] and the Dermoscopy Atlas website [23]. From the ISIC Archive, images with a melanoma thickness label and images from the category melanoma in situ were downloaded with corresponding metadata using the provided interface. The rest of the data was scraped from the Dermoscopy Atlas website. For scraped images, the thickness label was extracted from the image description or the assigned diagnosis of melanoma in situ. Initially, we collected 1290 images with matching Breslow thickness.

2.2. T-Category Label

Online image repositories are based on community contribution, which causes the issue of potentially duplicated images appearing in the same repository and the same image being present in multiple repositories. Possible overlap between our data and the datasets commonly used for training was also assessed. Most skin lesion diagnosis methods were trained on ISIC datasets from the years 2016–2020 [24,25,26,27,28,29], so we removed overlapping images between our benchmark dataset and ISIC challenge datasets to ensure a fair comparison.

Following [30], we found duplicates using the czkawka [31] application that compares image hashes using hamming distance to find identical and similar images. The search was performed using the Lanczos3 resize algorithm and gradient type hashes of size 16. The similarity threshold was set to 40 to find images with even minimal resemblance. All flagged duplicates were manually investigated to exclude false positives. We do not consider multiple images of the same lesion as duplicates. We began by identifying duplicate images within the collected dataset, where we found and removed 7 identical images. Next, we searched for duplicates between the ISIC Challenges and our benchmark datasets. Initially, this search was based on ISIC IDs, which led to us identifying and removing 146 duplicate images. Additionally, using the czkawka tool, we detected 33 more duplicate images with different ISIC IDs.

Currently, melanoma stages are defined based on metrics proposed by the American Joint Committee on Cancer (AJCC) using, among others, lesion thicknesses [32]. We assigned the labels based on the T-grouping of the AJCC staging method. Breslow thickness indicates tumor thickness from the granular layer of the epidermis to the deepest level of invasion [32]. We assigned T-category labels based on thickness information according to the Table 1. We grouped the a and b categories together as we did not have ulceration information to distinguish them.

Figure 1 presents the final composition of the dataset. A total of 1104 images with thickness information were collected. Of these, 907 images came from the ISIC archive, and 197 were obtained through web scraping from the Dermoscopy Atlas website. The dataset includes 1017 dermoscopic images and 87 clinical photographs. Atlas data did not include the image type, so we labeled it based on visual characteristics observed in the data.

3. Results

To establish benchmark results on the EMB dataset, we tested various methods with code and model weights available online. We tested EfficientNet Ensemble, the winner of the SIIM-ISIC challenge in 2020 [33]; Class-Enhancement Contrastive Learning for Long-tailed Skin Lesion Classification (ECL) [34]; and deep mask pixel-wise supervision (DMP) [35]. All methods were trained on data from the ISIC challenge datasets without external data. These methods were selected based on the public availability of the code; trained models; and the demonstration of SOTA performance. Clinical images were excluded from testing as the methods were trained on dermoscopic images only. For the ISIC 2020 winner, we tested individual models from the ensemble, selecting only those that did not utilize metadata. For ECL, we tested two versions of the model: the 8-class model and the 9-class model. Similarly, for deep pixel-wise supervision, we tested the ConvNeXt 9-class models and the 2-class models. Input images for each model were preprocessed according to the specific requirements described in the original papers. Classification accuracy was assessed by checking whether the model correctly predicted the melanoma/malignant class. Since our dataset contains only melanoma cases, all images should be classified as either the positive class in binary models or the melanoma category in multi-class models.

Results achieved by tested models are shown in Table 2. Column All shows results for the entire dataset, and Columns Tis-T4 provide results for lesions in the corresponding thickness stages. Column Rep. shows the average accuracy reported by the authors on the ISIC dataset on which the model was tested.

All tested algorithms performed well on the datasets they were originally trained and evaluated on, but exhibited a significant drop in performance when tested on our dataset. The best-performing model from the ISIC winner ensemble (EfficientNet-B4 768 9c) reached an accuracy of 97.71% on the combined data from ISIC 2018-2020. However, when evaluated on our data, it achieved a median accuracy of 0.54 with a 95% confidence interval of $[eqn]$ . Similarly, the ECL 9 class model, which achieved an average accuracy of 86.11% on the ISIC 2019 test, showed a reduced accuracy of 0.40 (confidence interval $[eqn]$ ) on EMB data. The deep mask pixel-wise supervision (ConvNeXt-DMP 224 2c) achieved 90.70% accuracy on ISIC 2019+2020 data, and on EMB, its accuracy declined to 0.55 with a confidence interval of $[eqn]$ . All models showed the lowest accuracy on in situ and T1 melanomas, indicating difficulty in detecting early-stage lesions. Performance generally improved with increasing thickness, with T3 and T4 melanomas yielding the highest accuracy for most models, although some achieved their best results on T2 cases.

Figure 2 shows a Uniform Manifold Approximation and Projection (UMAP) visualization of features extracted from melanoma images from the ISIC 2019 dataset (blue) and from EMB (orange). UMAP allows for the visualization of high-dimensional features on a 2D plane, which allows for the observation of the distribution and relationships between data points. For the well-trained model, one would expect a homogeneous and compact feature representation. In the case of the presented models, the visualization reveals a relatively uniform structure in feature representation for both datasets, yet non-uniformity exists between the challenge data and the benchmark data. This indicates that while early-stage melanoma shares some common features with later-stage melanoma, these shared features are insufficient for accurate classification when using models trained on the ISIC Challenge datasets.

Figure 3 shows Grad-CAM [36] visualizations of activation maps from the last convolutional layer for three models. Grad-CAM visualizations offer insights into the decision-making processes of the tested deep learning models by showing regions that are important for the prediction. Upon examining the images, it can be observed that models generally focus on the lesion areas, particularly in stages T2 to T4. However, in stages Tis and T1 cases, the models often concentrate on seemingly unrelated regions while ignoring areas with high pigmentation.

4. Limitations and Future Work

While our dataset serves as a benchmark for early melanoma diagnosis, several limitations should be considered. Although efforts were made to separate the benchmark dataset from the ISIC Challenge sets, some overlap may remain. Certain images in the benchmark dataset may appear in training sets if they originate from the ISIC Archive. This could affect benchmarking reliability, as models might previously be exposed to similar data in the training and validation stages. In addition, the dataset contains a higher representation of thin and in situ melanoma cases compared to more advanced stages. There is also a larger proportion of dermoscopic images relative to clinical images. The dataset prioritizes early intervention research, so we did not concentrate on balancing the dataset but rather on providing the highest possible number of melanoma images. Also, the dataset’s composition is derived from public repositories, inherently reflecting the “as-is” nature of available data, and thus was not specifically curated to mitigate a racial bias.

Future work will focus on developing a specialized model for melanoma T-category classification. This model will label data with an unknown T-category, which will then be used to oversample T1, T1, and T2 data in the ISIC training dataset. This strategy aims to counteract the underrepresentation of thin lesions, thereby improving the accuracy of deep learning models in identifying early melanoma.

5. Conclusions

Deep learning algorithms have been developed to support non-dermatologist clinicians or assist in identifying suspicious lesions prior to clinical care. While significant progress has been made in this field, several key challenges remain unresolved. Our analysis highlights the following issues for deep learning algorithms in detecting malignant melanoma: (1) state-of-the-art algorithms exhibit significantly worse performance on in situ and T1 melanoma images compared to their reported performance on the ISIC Challenge dataset; (2) this results in a substantial drop in performance when evaluated on EMB dataset, which includes mainly early-stage melanoma; and (3) clinical images pose a notable challenge for the tested models, further impacting their effectiveness. A lack of diversity in the training data likely contributes to the described problems. The unknown distribution of melanoma stages and the underrepresentation of early-stage lesions in commonly used datasets introduce biases, leading to early-stage melanoma often being misclassified as benign. Models struggle to effectively transfer their learned features to new data with early-stage cases. This suggests that the visual characteristics of early melanoma are not adequately learned as discriminative features when broader, less-specific datasets are used for training. Misclassification of Tis and T1 category melanomas is a performance deficit that poses a considerable barrier to the clinical applicability of these automated systems, as reliable early detection is critical for improving patient outcomes. Most of the currently developed models use the same data for training and rarely perform a cross-dataset evaluation of model performance. Addressing these challenges is crucial for developing more robust and reliable deep learning systems for melanoma detection. The Early-Stage Melanoma Benchmark dataset provides a valuable resource to help tackle these issues by enabling T-category-specific analysis and cross-dataset evaluation. Through its curated T-category information, it enables testing of model performance on melanomas with varying thicknesses. This ultimately supports the development of automated diagnostic systems that are more robust, clinically relevant, and effective in detecting early-stage melanoma.