Novel Scoring Scale for Quality Assessment of Lung Ultrasound in the Emergency Department

TL;DR

This study introduces a new lung-specific scoring system for ultrasound quality assessment in emergency departments, which improves agreement between reviewers compared to the current general system.

Contribution

A novel, organ-specific scoring system for lung ultrasound quality assessment with better interobserver agreement than existing general systems.

Findings

The novel grading scale showed significantly higher interobserver agreement (ICC 0.703) compared to the ACEP scale (ICC 0.552).

The variance in scores was similar between the two scales, indicating consistent scoring spread.

The new system could improve ultrasound education by providing more consistent feedback.

Abstract

The use of a reliable scoring system for quality assessment (QA) is imperative to limit inconsistencies in measuring ultrasound acquisition skills. The current grading scale used for QA endorsed by the American College of Emergency Physicians (ACEP) is non-specific, applies irrespective of the type of study performed, and has not been rigorously validated. Our goal in this study was to determine whether a succinct, organ-specific grading scale designed for lung-specific QA would be more precise with better interobserver agreement.

This was a prospective validation study of an objective QA scale for lung ultrasound (LUS) in the emergency department. We identified the first 100 LUS performed in normal clinical practice in the year 2020. Four reviewers at an urban academic center who were either emergency ultrasound fellowship-trained or current fellows with at least six months of QA experience scored each study, resulting in a total of 400. The primary outcome was the level of agreement between the reviewers. Our secondary outcome was the variability of the scores given to the studies. For the agreement between reviewers, we computed the intraclass correlation coefficient (ICC) based on a two-way random-effect model with a single rater for each grading scale. We generated 10,000 bootstrapped ICCs to construct 95% confidence intervals (CI) for both grading systems. A two-sided one-sample t-test was used to determine whether there were differences in the bootstrapped ICCs between the two grading systems.

The ICC between reviewers was 0.552 (95% CI 0.40–0.68) for the ACEP grading scale and 0.703 (95% CI 0.59–0.79) for the novel grading scale (P < 0.001), indicating significantly more interobserver agreement using the novel scale compared to the ACEP scale. The variance of scores was similar (0.93 and 0.92 for the novel and ACEP scales, respectively).

We found an increased interobserver agreement between reviewers when using the novel, organ-specific scale when compared with the ACEP grading scale. Increased consistency in feedback based on objective criteria directed to the specific, targeted organ provides an opportunity to enhance learner education and satisfaction with their ultrasound education.

Figures