Comparative Analysis of AI Models in Predicting Treatment Strategies for Unruptured Intracranial Aneurysms

TL;DR

This study compares AI models like ChatGPT-4 with expert decisions for treating unruptured brain aneurysms, finding that ChatGPT-4 performs best but experts remain essential.

Contribution

The study evaluates how well AI models align with expert neurovascular board decisions for unruptured intracranial aneurysm treatment strategies.

Findings

ChatGPT-4 showed highest accuracy (89%) in predicting conservative or operative management decisions.

ChatGPT-4 also had 73% accuracy in recommending specific treatment types for aneurysms.

AI models suggested shorter follow-up intervals for conservative management compared to the neurovascular board.

Abstract

Objectives: The increasing incidence of unruptured intracranial aneurysms (UIAs) has led to significant demands on neurovascular boards. Large language models (LLMs), such as ChatGPT-4, ChatGPT-3.5, Claude, and Atlas GPT, have emerged as tools to support clinical decision-making. This study compares treatment recommendations from these AI models with those of an interdisciplinary neurovascular board to evaluate their accuracy and alignment. Methods: We retrospectively included all 57 patients with UIAs discussed by the neurovascular board in 2023. The board’s consensus decision served as the reference standard. Key clinical and radiographic data, including PHASES, ELAPSS, and UIATS scores, were provided to the AI models. Each model was tasked with recommending either conservative or operative management and specifying the treatment modality (clipping, coiling, flow diverter, or WEB device/flow diverter) where appropriate. AI model recommendations were compared with the board’s decisions for management and the specific treatment modality of the UIA. Results: ChatGPT-4 achieved the highest accuracy in correctly predicting conservative or operative management (89%) and specific treatment types (73%), followed by Atlas GPT (74% accuracy in conservative/operative decisions and 55% accuracy in specific treatment types), Claude (70% accuracy in conservative/operative decisions and 50% accuracy in specific treatment types), and ChatGPT-3.5 (82% accuracy in conservative/operative decisions and 27% accuracy in specific treatment types). ChatGPT-3.5 displayed a strong preference for clipping (94.3%). ELAPSS scores significantly influenced AI recommendations and decision-making, particularly for ChatGPT-4 and ChatGPT-3.5. Follow-up recommendations for conservative management were shorter among AI models, with Claude suggesting the shortest interval (7.72 months) compared to the neurovascular board’s 13.36 months. Conclusions: AI models, particularly ChatGPT-4, align closely with expert neurovascular board decisions and offer promising support for initial clinical decision-making, particularly in resource-limited settings. However, interdisciplinary neurovascular boards remain unreplaceable for UIA management, and AI should be viewed as a complementary tool. The observed improvement from ChatGPT-3.5 to ChatGPT-4 underscores the rapid evolution of AI technology, and further advancements are expected to enhance both performance and accuracy in the future.

Figures