Deep Modeling and Optimization of Medical Image Classification

TL;DR

This paper explores advanced deep learning models, including CLIP variants, federated learning, and traditional ML, to improve medical image classification accuracy while addressing data privacy and generalization challenges.

Contribution

It introduces a novel CLIP variant with multiple CNNs and ViTs, combines deep models with federated learning for privacy, and integrates traditional ML to enhance generalization in medical image classification.

Findings

MaxViT achieves 87.03% accuracy on HAM10000 with multimodal learning.

ConvNeXt-L outperforms Swin-B with an F1-score of 83.98% in federated learning.

SVM improves metrics by approximately 2% for Swin transformer models.

Abstract

Deep models, such as convolutional neural networks (CNNs) and vision transformer (ViT), demonstrate remarkable performance in image classification. However, those deep models require large data to fine-tune, which is impractical in the medical domain due to the data privacy issue. Furthermore, despite the feasible performance of contrastive language image pre-training (CLIP) in the natural domain, the potential of CLIP has not been fully investigated in the medical field. To face these challenges, we considered three scenarios: 1) we introduce a novel CLIP variant using four CNNs and eight ViTs as image encoders for the classification of brain cancer and skin cancer, 2) we combine 12 deep models with two federated learning techniques to protect data privacy, and 3) we involve traditional machine learning (ML) methods to improve the generalization ability of those deep models in unseen domain data. The experimental results indicate that maxvit shows the highest averaged (AVG) test metrics (AVG = 87.03\%) in HAM10000 dataset with multimodal learning, while convnext\_l demonstrates remarkable test with an F1-score of 83.98\% compared to swin\_b with 81.33\% in FL model. Furthermore, the use of support vector machine (SVM) can improve the overall test metrics with AVG of $\sim 2\%$ for swin transformer series in ISIC2018. Our codes are available at https://github.com/AIPMLab/SkinCancerSimulation.