Comparative Analysis of Vision Transformers and Convolutional Neural Networks for Medical Image Classification

TL;DR

This study compares Vision Transformers and CNNs across three medical imaging tasks, revealing that different architectures excel depending on the specific application, thus guiding model selection in medical AI.

Contribution

It provides a comprehensive comparison of CNN and ViT architectures on multiple medical imaging tasks, highlighting task-specific performance differences.

Findings

ResNet-50 achieved 98.37% accuracy on chest X-ray classification

DeiT-Small excelled at brain tumor detection with 92.16% accuracy

EfficientNet-B0 led skin cancer classification at 81.84% accuracy

Abstract

The emergence of Vision Transformers (ViTs) has revolutionized computer vision, yet their effectiveness compared to traditional Convolutional Neural Networks (CNNs) in medical imaging remains under-explored. This study presents a comprehensive comparative analysis of CNN and ViT architectures across three critical medical imaging tasks: chest X-ray pneumonia detection, brain tumor classification, and skin cancer melanoma detection. We evaluated four state-of-the-art models - ResNet-50, EfficientNet-B0, ViT-Base, and DeiT-Small - across datasets totaling 8,469 medical images. Our results demonstrate task-specific model advantages: ResNet-50 achieved 98.37% accuracy on chest X-ray classification, DeiT-Small excelled at brain tumor detection with 92.16% accuracy, and EfficientNet-B0 led skin cancer classification at 81.84% accuracy. These findings provide crucial insights for practitioners selecting architectures for medical AI applications, highlighting the importance of task-specific architecture selection in clinical decision support systems.