Evaluating Large Vision-language Models for Surgical Tool Detection

TL;DR

This paper systematically evaluates large vision-language models for surgical tool detection, demonstrating that Qwen2.5 outperforms other models in zero-shot and fine-tuned settings, advancing AI's role in surgical scene understanding.

Contribution

It provides a comprehensive assessment of VLMs in surgical tool detection, highlighting Qwen2.5's superior performance and generalization capabilities over existing models.

Findings

Qwen2.5 achieves the best detection accuracy among evaluated VLMs.

Qwen2.5 shows strong zero-shot generalization compared to Grounding DINO.

Grounding DINO excels in localization tasks.

Abstract

Surgery is a highly complex process, and artificial intelligence has emerged as a transformative force in supporting surgical guidance and decision-making. However, the unimodal nature of most current AI systems limits their ability to achieve a holistic understanding of surgical workflows. This highlights the need for general-purpose surgical AI systems capable of comprehensively modeling the interrelated components of surgical scenes. Recent advances in large vision-language models that integrate multimodal data processing offer strong potential for modeling surgical tasks and providing human-like scene reasoning and understanding. Despite their promise, systematic investigations of VLMs in surgical applications remain limited. In this study, we evaluate the effectiveness of large VLMs for the fundamental surgical vision task of detecting surgical tools. Specifically, we investigate three state-of-the-art VLMs, Qwen2.5, LLaVA1.5, and InternVL3.5, on the GraSP robotic surgery dataset under both zero-shot and parameter-efficient LoRA fine-tuning settings. Our results demonstrate that Qwen2.5 consistently achieves superior detection performance in both configurations among the evaluated VLMs. Furthermore, compared with the open-set detection baseline Grounding DINO, Qwen2.5 exhibits stronger zero-shot generalization and comparable fine-tuned performance. Notably, Qwen2.5 shows superior instrument recognition, while Grounding DINO demonstrates stronger localization.