Inter-rater reliability of point-of-care ultrasound during out-of-hospital cardiac arrest: an ancillary analysis of the observational prospective ACE trial

TL;DR

Emergency physicians and experts disagree on ultrasound images during cardiac arrest, but this doesn't affect outcome predictions.

Contribution

This study evaluates inter-rater reliability of point-of-care ultrasound during out-of-hospital cardiac arrest.

Findings

Interrater agreement between emergency physicians and experts was low (κ = 0.25–0.26).

Inter-expert agreement was higher (κ = 0.75).

Discrepancies in image interpretation did not significantly affect ROSC prognostication.

Abstract

Prognostication in out-of-hospital cardiac arrest (OHCA) remains challenging. While point-of-care ultrasound (POCUS) has demonstrated utility in identifying reversible causes and predicting outcomes, the quality and reliability of echocardiographic assessments in this context remain poorly characterized. This study aimed to evaluate interrater agreement between trained emergency physicians (POCUS operators) and ultrasound experts in assessing cardiac motion on ultrasound during cardiopulmonary resuscitation (CPR).

This study was an ancillary analysis of the ACE trial, a multicenter, prospective, observational study conducted between November 2018 and January 2023, which included 293 patients. A random sample comprising 20 % of recorded ultrasound cine loops was included in the present analysis. Two independent experts evaluated the presence of visible cardiac motion. Their assessments were compared with those of the POCUS operator using Cohen’s κ coefficients.

A total of 52 POCUS cine loops were collected. The median patient age was 69 years. The presumed etiology of OHCA was cardiac in 61.5 % of cases, noncardiac medical in 32.7 %, and traumatic in 5.8 %. Expert 1 excluded 16 loops (31 %) and Expert 2 excluded 9 loops (17 %) because of insufficient image quality. The κ coefficients for detection of cardiac motion were 0.26 (95 % CI, −0.05–0.58) and 0.25 (95 % CI, −0.05–0.54) for agreement between operators and experts 1 and 2, respectively. Inter-expert agreement was higher, with a κ of 0.75 (95 % CI, 0.51–0.98). The positive predictive value of cardiac standstill for absence of return of spontaneous circulation (ROSC) did not differ significantly between operators and experts (74.3 % vs 65.0 % and 72.4 %; p = 0.47 and 0.87, respectively).

Agreement between emergency physicians and experts regarding POCUS image quality and interpretation of cardiac motion during OHCA was limited. However, this discrepancy did not appear to significantly affect the prognostication of ROSC. Further training and standardization of image acquisition and interpretation criteria may improve POCUS reliability in this setting.

Trial registration: This paper is an ancillary study of the ACE trial, registered on ClinicalTrials.gov (Identifier: NCT 03494153) on March 29, 2018.

Figures