Bringing AI pipelines onto cloud-HPC: setting a baseline for accuracy of COVID-19 AI diagnosis

TL;DR

This paper presents a portable, scalable AI pipeline on HPC for COVID-19 diagnosis from CT scans, establishing a performance baseline and identifying a high-accuracy DNN model.

Contribution

It introduces the CLAIRE COVID-19 Universal Pipeline using StreamFlow for reproducible, optimized AI workflows on HPC for COVID-19 detection.

Findings

Achieved over 90% accuracy in COVID-19 lesion detection.

Demonstrated the portability and scalability of AI pipelines on HPC.

Established a performance baseline for AI-based COVID-19 diagnosis.

Abstract

HPC is an enabling platform for AI. The introduction of AI workloads in the HPC applications basket has non-trivial consequences both on the way of designing AI applications and on the way of providing HPC computing. This is the leitmotif of the convergence between HPC and AI. The formalized definition of AI pipelines is one of the milestones of HPC-AI convergence. If well conducted, it allows, on the one hand, to obtain portable and scalable applications. On the other hand, it is crucial for the reproducibility of scientific pipelines. In this work, we advocate the StreamFlow Workflow Management System as a crucial ingredient to define a parametric pipeline, called "CLAIRE COVID-19 Universal Pipeline," which is able to explore the optimization space of methods to classify COVID-19 lung lesions from CT scans, compare them for accuracy, and therefore set a performance baseline. The universal pipeline automatizes the training of many different Deep Neural Networks (DNNs) and many different hyperparameters. It, therefore, requires a massive computing power, which is found in traditional HPC infrastructure thanks to the portability-by-design of pipelines designed with StreamFlow. Using the universal pipeline, we identified a DNN reaching over 90% accuracy in detecting COVID-19 lesions in CT scans.