Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Dataset Labeling

TL;DR

This study demonstrates that a gamified crowdsourcing platform can produce lung ultrasound labels of comparable quality to clinical experts, offering a scalable solution for creating high-quality datasets for machine learning.

Contribution

The paper introduces a gamified crowdsourcing method with quality control that achieves expert-level labeling accuracy for lung ultrasound data.

Findings

Crowdsourced labels achieved 87.9% concordance with expert standards.

Crowd labels outperformed individual experts when compared to reference standards.

The approach is scalable and maintains high label quality for medical imaging datasets.

Abstract

Study Objective: Machine learning models have advanced medical image processing and can yield faster, more accurate diagnoses. Despite a wealth of available medical imaging data, high-quality labeled data for model training is lacking. We investigated whether a gamified crowdsourcing platform enhanced with inbuilt quality control metrics can produce lung ultrasound clip labels comparable to those from clinical experts. Methods: 2,384 lung ultrasound clips were retrospectively collected from 203 patients. Six lung ultrasound experts classified 393 of these clips as having no B-lines, one or more discrete B-lines, or confluent B-lines to create two sets of reference standard labels (195 training set clips and 198 test set clips). Sets were respectively used to A) train users on a gamified crowdsourcing platform, and B) compare concordance of the resulting crowd labels to the concordance of individual experts to reference standards. Results: 99,238 crowdsourced opinions on 2,384 lung ultrasound clips were collected from 426 unique users over 8 days. On the 198 test set clips, mean labeling concordance of individual experts relative to the reference standard was 85.0% +/- 2.0 (SEM), compared to 87.9% crowdsourced label concordance (p=0.15). When individual experts' opinions were compared to reference standard labels created by majority vote excluding their own opinion, crowd concordance was higher than the mean concordance of individual experts to reference standards (87.4% vs. 80.8% +/- 1.6; p<0.001). Conclusion: Crowdsourced labels for B-line classification via a gamified approach achieved expert-level quality. Scalable, high-quality labeling approaches may facilitate training dataset creation for machine learning model development.