Explainable Knowledge Distillation for Efficient Medical Image Classification

TL;DR

This paper presents an explainable knowledge distillation framework that trains compact, efficient, and interpretable models for COVID-19 and lung cancer classification from chest X-ray images, balancing accuracy and computational cost.

Contribution

It introduces a hybrid supervision approach using high-capacity teachers and a lightweight student model, with explainability via Score-CAM visualizations, for medical image classification.

Findings

Student models achieve high accuracy with fewer parameters.

Distilled models have faster inference suitable for clinical settings.

Explainability insights improve trust in model decisions.

Abstract

This study comprehensively explores knowledge distillation frameworks for COVID-19 and lung cancer classification using chest X-ray (CXR) images. We employ high-capacity teacher models, including VGG19 and lightweight Vision Transformers (Visformer-S and AutoFormer-V2-T), to guide the training of a compact, hardware-aware student model derived from the OFA-595 supernet. Our approach leverages hybrid supervision, combining ground-truth labels with teacher models' soft targets to balance accuracy and computational efficiency. We validate our models on two benchmark datasets: COVID-QU-Ex and LCS25000, covering multiple classes, including COVID-19, healthy, non-COVID pneumonia, lung, and colon cancer. To interpret the spatial focus of the models, we employ Score-CAM-based visualizations, which provide insight into the reasoning process of both teacher and student networks. The results demonstrate that the distilled student model maintains high classification performance with significantly reduced parameters and inference time, making it an optimal choice in resource-constrained clinical environments. Our work underscores the importance of combining model efficiency with explainability for practical, trustworthy medical AI solutions.