VS-Assistant: Versatile Surgery Assistant on the Demand of Surgeons

TL;DR

The paper introduces VS-Assistant, a versatile surgical AI that understands surgeon intentions and performs multiple tasks like scene analysis and instrument detection, improving surgical support through advanced multimodal models.

Contribution

It develops a multimodal large language model with a novel mixture of projectors and function-calling tuning to enable versatile, accurate surgical understanding and task execution.

Findings

Outperforms existing models in surgical scene understanding

Accurately detects and segments surgical instruments

Enhances textual and visual analysis in neurosurgery data

Abstract

The surgical intervention is crucial to patient healthcare, and many studies have developed advanced algorithms to provide understanding and decision-making assistance for surgeons. Despite great progress, these algorithms are developed for a single specific task and scenario, and in practice require the manual combination of different functions, thus limiting the applicability. Thus, an intelligent and versatile surgical assistant is expected to accurately understand the surgeon's intentions and accordingly conduct the specific tasks to support the surgical process. In this work, by leveraging advanced multimodal large language models (MLLMs), we propose a Versatile Surgery Assistant (VS-Assistant) that can accurately understand the surgeon's intention and complete a series of surgical understanding tasks, e.g., surgical scene analysis, surgical instrument detection, and segmentation on demand. Specifically, to achieve superior surgical multimodal understanding, we devise a mixture of projectors (MOP) module to align the surgical MLLM in VS-Assistant to balance the natural and surgical knowledge. Moreover, we devise a surgical Function-Calling Tuning strategy to enable the VS-Assistant to understand surgical intentions, and thus make a series of surgical function calls on demand to meet the needs of the surgeons. Extensive experiments on neurosurgery data confirm that our VS-Assistant can understand the surgeon's intention more accurately than the existing MLLM, resulting in overwhelming performance in textual analysis and visual tasks. Source code and models will be made public.