VS-Assistant: Versatile Surgery Assistant on the Demand of Surgeons
Zhen Chen, Xingjian Luo, Jinlin Wu, Danny T.M. Chan, Zhen Lei, Jinqiao, Wang, Sebastien Ourselin, Hongbin Liu

TL;DR
The paper introduces VS-Assistant, a versatile surgical AI that understands surgeon intentions and performs multiple tasks like scene analysis and instrument detection, improving surgical support through advanced multimodal models.
Contribution
It develops a multimodal large language model with a novel mixture of projectors and function-calling tuning to enable versatile, accurate surgical understanding and task execution.
Findings
Outperforms existing models in surgical scene understanding
Accurately detects and segments surgical instruments
Enhances textual and visual analysis in neurosurgery data
Abstract
The surgical intervention is crucial to patient healthcare, and many studies have developed advanced algorithms to provide understanding and decision-making assistance for surgeons. Despite great progress, these algorithms are developed for a single specific task and scenario, and in practice require the manual combination of different functions, thus limiting the applicability. Thus, an intelligent and versatile surgical assistant is expected to accurately understand the surgeon's intentions and accordingly conduct the specific tasks to support the surgical process. In this work, by leveraging advanced multimodal large language models (MLLMs), we propose a Versatile Surgery Assistant (VS-Assistant) that can accurately understand the surgeon's intention and complete a series of surgical understanding tasks, e.g., surgical scene analysis, surgical instrument detection, and segmentation…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNursing Roles and Practices · Global Health Workforce Issues · Medical Education and Admissions
MethodsALIGN
