Resetting the baseline: CT-based COVID-19 diagnosis with Deep Transfer Learning is not as accurate as widely thought

TL;DR

This paper critically evaluates the effectiveness of deep transfer learning for CT-based COVID-19 diagnosis, revealing that many published results overestimate performance due to data curation issues and emphasizing the need for transparent evaluation.

Contribution

The study systematically analyzes 14 state-of-the-art models across 200 training sessions, exposing overestimations and providing realistic baselines for COVID-19 detection with CT images.

Findings

Many published results overestimate transfer learning performance

Inappropriate data curation leads to inflated accuracy claims

Realistic scenarios and transparent baselines are essential for evaluation

Abstract

Deep learning is gaining instant popularity in computer aided diagnosis of COVID-19. Due to the high sensitivity of Computed Tomography (CT) to this disease, CT-based COVID-19 detection with visual models is currently at the forefront of medical imaging research. Outcomes published in this direction are frequently claiming highly accurate detection under deep transfer learning. This is leading medical technologists to believe that deep transfer learning is the mainstream solution for the problem. However, our critical analysis of the literature reveals an alarming performance disparity between different published results. Hence, we conduct a systematic thorough investigation to analyze the effectiveness of deep transfer learning for COVID-19 detection with CT images. Exploring 14 state-of-the-art visual models with over 200 model training sessions, we conclusively establish that the published literature is frequently overestimating transfer learning performance for the problem, even in the prestigious scientific sources. The roots of overestimation trace back to inappropriate data curation. We also provide case studies that consider more realistic scenarios, and establish transparent baselines for the problem. We hope that our reproducible investigation will help in curbing hype-driven claims for the critical problem of COVID-19 diagnosis, and pave the way for a more transparent performance evaluation of techniques for CT-based COVID-19 detection.