Surgical-LLaVA: Toward Surgical Scenario Understanding via Large Language and Vision Models

TL;DR

Surgical-LLaVA is a specialized large vision-language model designed to understand and interact with surgical images and videos, enhancing multi-modal communication in surgical contexts.

Contribution

The paper introduces Surgical-LLaVA, a novel LVLM tailored for surgical scenarios, integrating visual data into language models and fine-tuning on surgical instruction data.

Findings

Demonstrates impressive multi-modal chat abilities in surgical contexts

Achieves superior performance on surgical visual question-answering datasets

Displays potential for handling complex surgical scenarios

Abstract

Conversation agents powered by large language models are revolutionizing the way we interact with visual data. Recently, large vision-language models (LVLMs) have been extensively studied for both images and videos. However, these studies typically focus on common scenarios. In this work, we introduce an LVLM specifically designed for surgical scenarios. We integrate visual representations of surgical images and videos into the language feature space. Consequently, we establish a LVLM model, Surgical-LLaVA, fine-tuned on instruction following data of surgical scenarios. Our experiments demonstrate that Surgical-LLaVA exhibits impressive multi-modal chat abilities in surgical contexts, occasionally displaying multi-modal behaviors on unseen instructions. We conduct a quantitative evaluation of visual question-answering datasets for surgical scenarios. The results show superior performance compared to previous works, indicating the potential of our model to tackle more complex surgery scenarios.