# Assessing workflow impact and clinical utility of AI-assisted brain aneurysm detection: A multi-reader study

## TL;DR

This study shows that AI assistance for brain aneurysm detection did not improve radiologists' accuracy and actually increased reading times.

## Contribution

The study provides empirical evidence on the real-world impact of AI tools on radiologists' workflow and diagnostic performance.

## Key findings

- AI assistance did not significantly improve sensitivity for either junior or senior radiologists.
- Reading times increased significantly when using AI assistance for both reader groups.
- Radiologists' confidence levels remained unchanged with or without AI assistance.

## Abstract

•AI assistance did not significantly improve sensitivity for the radiologists.•Integration of AI tools requires thorough clinical validation for real-world use.•AI assistance increased reading times for both junior and senior radiologists.

AI assistance did not significantly improve sensitivity for the radiologists.

Integration of AI tools requires thorough clinical validation for real-world use.

AI assistance increased reading times for both junior and senior radiologists.

Despite the plethora of AI-based algorithms developed for anomaly detection in radiology, subsequent integration into clinical setting is rarely evaluated. In this work, we assess the applicability and utility of an AI-based model for brain aneurysm detection comparing the performance of two readers with different levels of experience (2 and 13 years). We aim to answer the following questions: 1) Do the readers improve their performance when assisted by the AI algorithm? 2) How much does the AI algorithm impact routine clinical workflow? We reuse and enlarge our open-access, Time-Of-Flight Magnetic Resonance Angiography dataset (N = 460). We use 360 subjects for training/validating our algorithm and 100 as unseen test set for the reading session. Even though our model reaches state-of-the-art results on the test set (sensitivity = 74 %, false positive rate = 1.6), we show that neither the junior nor the senior reader significantly increase their sensitivity (p = 0.59, p = 1, respectively). In addition, we find that reading time for both readers is significantly higher in the “AI-assisted” setting than in the “Unassisted” (+15 s, on average; p=3×10-4 junior, p=3×10-5 senior). The confidence reported by the readers is unchanged across the two settings, indicating that the AI assistance does not influence the certainty of the diagnosis. Our findings highlight the importance of clinical validation of AI algorithms in a clinical setting involving radiologists. This study should serve as a reminder to the community to always examine the real-word effectiveness and workflow impact of proposed algorithms.