minoHealth.ai: A Clinical Evaluation Of Deep Learning Systems For the Diagnosis of Pleural Effusion and Cardiomegaly In Ghana, Vietnam and the United States of America

TL;DR

This study evaluates the performance of minoHealth.ai's deep learning systems in diagnosing cardiomegaly and pleural effusion from chest X-rays across Ghana, Vietnam, and the USA, comparing AI accuracy with experienced radiologists.

Contribution

It provides a cross-country clinical evaluation of AI diagnostic systems, demonstrating their superior performance over radiologists in detecting specific thoracic conditions.

Findings

AI systems achieved higher AUC-ROC scores than radiologists.

AI outperformed radiologists by approximately 10% in accuracy.

The study confirms AI's potential for rapid, accurate diagnosis in diverse settings.

Abstract

A rapid and accurate diagnosis of cardiomegaly and pleural effusion is of the utmost importance to reduce mortality and medical costs. Artificial Intelligence has shown promise in diagnosing medical conditions. With this study, we seek to evaluate how well Artificial Intelligence (AI) systems, developed my minoHealth AI Labs, will perform at diagnosing cardiomegaly and pleural effusion, using chest x-rays from Ghana, Vietnam and the USA, and how well AI systems will perform when compared with radiologists working in Ghana. The evaluation dataset used in this study contained 100 images randomly selected from three datasets. The Deep Learning models were further tested on a larger Ghanaian dataset containing five hundred and sixty one (561) samples. Two AI systems were then evaluated on the evaluation dataset, whilst we also gave the same chest x-ray images within the evaluation dataset to 4 radiologists, with 5 - 20 years experience, to diagnose independently. For cardiomegaly, minoHealth-ai systems scored Area under the Receiver operating characteristic Curve (AUC-ROC) of 0.9 and 0.97 while the AUC-ROC of individual radiologists ranged from 0.77 to 0.87. For pleural effusion, the minoHealth-ai systems scored 0.97 and 0.91 whereas individual radiologists scored between 0.75 and 0.86. On both conditions, the best performing AI model outperforms the best performing radiologist by about 10%. We also evaluate the specificity, sensitivity, negative predictive value (NPV), and positive predictive value (PPV) between the minoHealth-ai systems and radiologists.