# Real-World Integration of an Automated Tool for Intracranial Hemorrhage Detection in an Unselected Cohort of Emergency Department Patients—An External Validation Study

## TL;DR

This study evaluated a deep learning tool for detecting brain bleeding in emergency patients and found it had high false positives and lower accuracy than a first-year radiology resident.

## Contribution

The study provides real-world validation of a DL tool for ICH detection using an unselected ED cohort and compares it to a novice radiologist.

## Key findings

- RAPID ICH had 87.3% sensitivity and 74% specificity for detecting ICH.
- The tool had 203 false positives and 8 false negatives in 844 cases.
- A first-year resident outperformed RAPID ICH in both sensitivity and specificity.

## Abstract

Background/Objectives: Intracranial hemorrhage (ICH) is a life-threatening condition that can be rapidly detected by non-contrast head computed tomography (NCCT). RAPID ICH is a deep learning (DL) tool for automatic ICH identification using NCCT. Our aim was to assess the real-world performance of RAPID ICH compared to that of a first-year radiology resident on consecutively acquired NCCTs from patients referred from the Emergency Department. Methods: This single-center retrospective cohort study included NCCTs acquired on the same CT scanner over three months. Exclusion criteria were motion or metallic artifacts that substantially degraded the NCCT quality and incomplete NCCTs. Two senior neuroradiologists conducted ground-truth labeling of the NCCTs regarding ICH presence in a binary manner. The first-year radiology resident assessed NCCTs for ICH presence and was blinded to the ground-truth labeling. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were computed for the RAPID ICH identifications and for the first-year radiology resident’s ICH identifications. Results: After applying exclusion criteria, 844 NCCTs remained. Ground-truth labeling found ICH in 63 NCCTs. RAPID ICH showed 87.3% sensitivity, 74% specificity, 21.3% PPV, and 98.6% NPV, while the first-year radiology resident achieved 95.2% sensitivity, 90.8% specificity, 45.5% PPV, and 99.6% NPV. There were 8 false-negative and 203 false-positive RAPID ICH identifications. Conclusions: RAPID ICH’s sensitivity and specificity were lower than in prior studies performed using RAPID ICH, and there was a high number of false-positive RAPID ICH identifications, limiting the generalizability of the assessed version of this DL tool. Testing DL tools by comparing them with radiologists of varying experience can provide valuable insights into their performance.