PatchAlign:Fair and Accurate Skin Disease Image Classification by Alignment with Clinical Labels

TL;DR

PatchAlign is a novel method that improves skin disease image classification accuracy and fairness across skin tones by aligning image features with clinical labels using graph optimal transport, even with limited data.

Contribution

It introduces PatchAlign, a new approach utilizing Graph Optimal Transport Loss and a learnable Masked GOTT to enhance fairness and accuracy in skin disease classification.

Findings

Improves accuracy by up to 6.2% over state-of-the-art methods.

Enhances fairness of true positive rates across skin tones.

Robust generalization across different skin types with limited samples.

Abstract

Deep learning models have achieved great success in automating skin lesion diagnosis. However, the ethnic disparity in these models' predictions needs to be addressed before deploying them. We introduce a novel approach, PatchAlign, to enhance skin condition image classification accuracy and fairness by aligning with clinical text representations of skin conditions. PatchAlign uses Graph Optimal Transport (GOT) Loss as a regularizer to perform cross-domain alignment. The representations obtained are robust and generalize well across skin tones, even with limited training samples. To reduce the effect of noise and artifacts in clinical dermatology images, we propose a learnable Masked Graph Optimal Transport for cross-domain alignment that further improves fairness metrics. We compare our model to the state-of-the-art FairDisCo on two skin lesion datasets with different skin types: Fitzpatrick17k and Diverse Dermatology Images (DDI). PatchAlign enhances the accuracy of skin condition image classification by 2.8% (in-domain) and 6.2% (out-domain) on Fitzpatrick17k, and 4.2% (in-domain) on DDI compared to FairDisCo. Additionally, it consistently improves the fairness of true positive rates across skin tones. The source code for the implementation is available at the following GitHub repository: https://github.com/aayushmanace/PatchAlign24, enabling easy reproduction and further experimentation.