How to Count Coughs: An Event-Based Framework for Evaluating Automatic Cough Detection Algorithm Performance

TL;DR

This paper introduces an event-based evaluation framework for cough detection algorithms, aligning performance metrics with clinical relevance and addressing limitations of traditional sample-based metrics.

Contribution

It proposes a novel event-based evaluation method for cough detection, providing open-source tools and guidelines to improve clinical applicability of ML algorithms.

Findings

Traditional metrics are biased by class imbalance.

Event-based metrics better reflect clinical cough counting needs.

Open-source framework facilitates standardized evaluation.

Abstract

Chronic cough disorders are widespread and challenging to assess because they rely on subjective patient questionnaires about cough frequency. Wearable devices running Machine Learning (ML) algorithms are promising for quantifying daily coughs, providing clinicians with objective metrics to track symptoms and evaluate treatments. However, there is a mismatch between state-of-the-art metrics for cough counting algorithms and the information relevant to clinicians. Most works focus on distinguishing cough from non-cough samples, which does not directly provide clinically relevant outcomes such as the number of cough events or their temporal patterns. In addition, typical metrics such as specificity and accuracy can be biased by class imbalance. We propose using event-based evaluation metrics aligned with clinical guidelines on significant cough counting endpoints. We use an ML classifier to illustrate the shortcomings of traditional sample-based accuracy measurements, highlighting their variance due to dataset class imbalance and sample window length. We also present an open-source event-based evaluation framework to test algorithm performance in identifying cough events and rejecting false positives. We provide examples and best practice guidelines in event-based cough counting as a necessary first step to assess algorithm performance with clinical relevance.