SurgMLLMBench: A Multimodal Large Language Model Benchmark Dataset for Surgical Scene Understanding

TL;DR

SurgMLLMBench introduces a comprehensive multimodal benchmark dataset for surgical scene understanding, combining pixel-level segmentation and VQA annotations across various surgical domains to facilitate development and evaluation of interactive surgical AI models.

Contribution

The paper presents SurgMLLMBench, a unified multimodal benchmark with new datasets and annotations, enabling consistent evaluation of surgical scene understanding models beyond traditional VQA tasks.

Findings

Models trained on SurgMLLMBench perform well across domains.

The benchmark supports richer visual-conversational interactions.

The dataset enhances reproducible evaluation in surgical AI research.

Abstract

Recent advances in multimodal large language models (LLMs) have highlighted their potential for medical and surgical applications. However, existing surgical datasets predominantly adopt a Visual Question Answering (VQA) format with heterogeneous taxonomies and lack support for pixel-level segmentation, limiting consistent evaluation and applicability. We present SurgMLLMBench, a unified multimodal benchmark explicitly designed for developing and evaluating interactive multimodal LLMs for surgical scene understanding, including the newly collected Micro-surgical Artificial Vascular anastomosIS (MAVIS) dataset. It integrates pixel-level instrument segmentation masks and structured VQA annotations across laparoscopic, robot-assisted, and micro-surgical domains under a unified taxonomy, enabling comprehensive evaluation beyond traditional VQA tasks and richer visual-conversational interactions. Extensive baseline experiments show that a single model trained on SurgMLLMBench achieves consistent performance across domains and generalizes effectively to unseen datasets. SurgMLLMBench will be publicly released as a robust resource to advance multimodal surgical AI research, supporting reproducible evaluation and development of interactive surgical reasoning models.