Leveraging Knowledge Distillation for Lightweight Skin Cancer Classification: Balancing Accuracy and Computational Efficiency

TL;DR

This paper introduces a knowledge distillation approach to develop a lightweight, high-accuracy skin cancer classifier suitable for resource-limited environments, combining model fusion, quantization, and rigorous training.

Contribution

It presents a novel fusion-based teacher model guiding a compact student model, achieving high accuracy with significantly reduced size for edge deployment.

Findings

Achieved 98.75% accuracy on HAM10000 dataset

Student model size reduced to 469.77 KB with 16-bit quantization

Model suitable for deployment on resource-constrained devices

Abstract

Skin cancer is a major concern to public health, accounting for one-third of the reported cancers. If not detected early, the cancer has the potential for severe consequences. Recognizing the critical need for effective skin cancer classification, we address the limitations of existing models, which are often too large to deploy in areas with limited computational resources. In response, we present a knowledge distillation based approach for creating a lightweight yet high-performing classifier. The proposed solution involves fusing three models, namely ResNet152V2, ConvNeXtBase, and ViT Base, to create an effective teacher model. The teacher model is then employed to guide a lightweight student model of size 2.03 MB. This student model is further compressed to 469.77 KB using 16-bit quantization, enabling smooth incorporation into edge devices. With six-stage image preprocessing, data augmentation, and a rigorous ablation study, the model achieves an impressive accuracy of 98.75% on the HAM10000 dataset and 98.94% on the Kaggle dataset in classifying benign and malignant skin cancers. With its high accuracy and compact size, our model appears to be a potential choice for accurate skin cancer classification, particularly in resource-constrained settings.