Sounds of COVID-19: exploring realistic performance of audio-based digital testing

TL;DR

This study evaluates the real-world effectiveness of audio-based COVID-19 detection using a large, bias-controlled dataset, demonstrating moderate accuracy and discussing practical integration into healthcare systems.

Contribution

It provides a comprehensive analysis of biases and methodological impacts on audio-based COVID-19 testing performance with a large crowdsourced dataset.

Findings

Unbiased model achieved AUC-ROC of 0.71

Biases and participant splits significantly affect performance

Discussion on clinical integration of audio-based testing

Abstract

Researchers have been battling with the question of how we can identify Coronavirus disease (COVID-19) cases efficiently, affordably and at scale. Recent work has shown how audio based approaches, which collect respiratory audio data (cough, breathing and voice) can be used for testing, however there is a lack of exploration of how biases and methodological decisions impact these tools' performance in practice. In this paper, we explore the realistic performance of audio-based digital testing of COVID-19. To investigate this, we collected a large crowdsourced respiratory audio dataset through a mobile app, alongside recent COVID-19 test result and symptoms intended as a ground truth. Within the collected dataset, we selected 5,240 samples from 2,478 participants and split them into different participant-independent sets for model development and validation. Among these, we controlled for potential confounding factors (such as demographics and language). The unbiased model takes features extracted from breathing, coughs, and voice signals as predictors and yields an AUC-ROC of 0.71 (95\% CI: 0.65$-$0.77). We further explore different unbalanced distributions to show how biases and participant splits affect performance. Finally, we discuss how the realistic model presented could be integrated in clinical practice to realize continuous, ubiquitous, sustainable and affordable testing at population scale.