NeuroABench: A Multimodal Evaluation Benchmark for Neurosurgical Anatomy Identification

TL;DR

NeuroABench is a new multimodal benchmark designed to evaluate neurosurgical anatomical understanding, revealing significant gaps in current models' performance compared to human experts, and highlighting the need for further development.

Contribution

This paper introduces NeuroABench, the first comprehensive multimodal benchmark for assessing anatomical comprehension in neurosurgery, with extensive annotated videos and a standardized evaluation framework.

Findings

Best MLLM achieves 40.87% accuracy in anatomical identification.

Neurosurgical trainees reach up to 56% accuracy, outperforming models.

Significant performance gap remains between models and human experts.

Abstract

Multimodal Large Language Models (MLLMs) have shown significant potential in surgical video understanding. With improved zero-shot performance and more effective human-machine interaction, they provide a strong foundation for advancing surgical education and assistance. However, existing research and datasets primarily focus on understanding surgical procedures and workflows, while paying limited attention to the critical role of anatomical comprehension. In clinical practice, surgeons rely heavily on precise anatomical understanding to interpret, review, and learn from surgical videos. To fill this gap, we introduce the Neurosurgical Anatomy Benchmark (NeuroABench), the first multimodal benchmark explicitly created to evaluate anatomical understanding in the neurosurgical domain. NeuroABench consists of 9 hours of annotated neurosurgical videos covering 89 distinct procedures and is developed using a novel multimodal annotation pipeline with multiple review cycles. The benchmark evaluates the identification of 68 clinical anatomical structures, providing a rigorous and standardized framework for assessing model performance. Experiments on over 10 state-of-the-art MLLMs reveal significant limitations, with the best-performing model achieving only 40.87% accuracy in anatomical identification tasks. To further evaluate the benchmark, we extract a subset of the dataset and conduct an informative test with four neurosurgical trainees. The results show that the best-performing student achieves 56% accuracy, with the lowest scores of 28% and an average score of 46.5%. While the best MLLM performs comparably to the lowest-scoring student, it still lags significantly behind the group's average performance. This comparison underscores both the progress of MLLMs in anatomical understanding and the substantial gap that remains in achieving human-level performance.