Explainable Melanoma Diagnosis with Contrastive Learning and LLM-based Report Generation

TL;DR

This paper introduces CEFM, a novel explainable melanoma diagnosis framework that uses contrastive learning and language models to improve interpretability and trust in deep learning models for clinical dermatology.

Contribution

The paper proposes a cross-modal framework that aligns clinical criteria with visual features and generates textual explanations, enhancing interpretability of melanoma classification models.

Findings

Achieved 92.79% accuracy and 0.961 AUC on public datasets.

Significant improvements in interpretability metrics.

Embeddings spatially align with clinical ABC criteria.

Abstract

Deep learning has demonstrated expert-level performance in melanoma classification, positioning it as a powerful tool in clinical dermatology. However, model opacity and the lack of interpretability remain critical barriers to clinical adoption, as clinicians often struggle to trust the decision-making processes of black-box models. To address this gap, we present a Cross-modal Explainable Framework for Melanoma (CEFM) that leverages contrastive learning as the core mechanism for achieving interpretability. Specifically, CEFM maps clinical criteria for melanoma diagnosis-namely Asymmetry, Border, and Color (ABC)-into the Vision Transformer embedding space using dual projection heads, thereby aligning clinical semantics with visual features. The aligned representations are subsequently translated into structured textual explanations via natural language generation, creating a transparent link between raw image data and clinical interpretation. Experiments on public datasets demonstrate 92.79% accuracy and an AUC of 0.961, along with significant improvements across multiple interpretability metrics. Qualitative analyses further show that the spatial arrangement of the learned embeddings aligns with clinicians' application of the ABC rule, effectively bridging the gap between high-performance classification and clinical trust.