# Melan-Dx: a knowledge-enhanced vision-language framework improves differential diagnosis of melanocytic neoplasm pathology

## TL;DR

Melan-Dx is an AI framework that improves the diagnosis of melanocytic neoplasm by combining pathology images with expert knowledge, enhancing accuracy in cancer classification.

## Contribution

Melan-Dx introduces a knowledge-enhanced vision-language framework for pathology diagnosis, avoiding fine-tuning of vision models while improving diagnostic accuracy.

## Key findings

- Melan-Dx achieves 0.869 accuracy for binary classification and 0.699 Top-1 accuracy among forty-class classification.
- It shows 23–70.6% improvement over zero-shot approaches and up to 8.4% better performance in whole slide image classification.
- The framework improves diagnostic accuracy without fine-tuning the vision backbone.

## Abstract

Melanoma is one of the top 5 cancer types, causes most deaths among skin cancers, and can be frequently misdiagnosed. Recent pathology image foundation models remain difficult to make accurate differential diagnosis across over forty melanocytic neoplasm histologic subtypes. Motivated by the diagnostic reasoning process of dermatopathologists, we curated a high-quality image and knowledge corpus database containing 2893 images and 1102 knowledge entries annotated by expert dermatopathologists at the University of Pennsylvania. Leveraging this multi-modal dataset, we present “Melan-Dx”, a knowledge-enhanced AI framework that augments frozen pathology vision-language models through retrieval from a curated vision-knowledge database, improving differential diagnosis at both patch and whole-slide levels. Melan-Dx, at its best performance, demonstrates 0.869 accuracy for binary classification, 0.699 Top-1 accuracy among forty-class classification, 0.915 ROC AUC for few-shot WSI tasks, and 0.925 AUPRC for fully supervised WSI tasks. Across all experimental settings, Melan-Dx shows improvements up to 13.8% over linear and fully finetuned methods, 23–70.6% over zero-shot approaches and up to 8.4% improvements in whole slide image classification. These findings suggest that a query database with a knowledge-enhanced AI framework can further improve existing pathology foundation models without fine-tuning the vision backbone. The code is publicly available at https://www.github.com/zhihuanglab/Melan-Dx-code.