Challenging Vision-Language Models with Surgical Data: A New Dataset and Broad Benchmarking Study

TL;DR

This study evaluates the capabilities of vision-language models in endoscopic surgery, revealing strengths in basic perception tasks but limitations in medical knowledge and highlighting the need for specialized model development.

Contribution

First large-scale benchmarking of VLMs on surgical data, comparing generalist and medical models across basic and advanced endoscopic tasks.

Findings

VLMs perform well on basic perception tasks like object counting.

Performance drops on tasks requiring medical knowledge.

Specialized medical VLMs underperform compared to generalist models.

Abstract

While traditional computer vision models have historically struggled to generalize to endoscopic domains, the emergence of foundation models has shown promising cross-domain performance. In this work, we present the first large-scale study assessing the capabilities of Vision Language Models (VLMs) for endoscopic tasks with a specific focus on laparoscopic surgery. Using a diverse set of state-of-the-art models, multiple surgical datasets, and extensive human reference annotations, we address three key research questions: (1) Can current VLMs solve basic perception tasks on surgical images? (2) Can they handle advanced frame-based endoscopic scene understanding tasks? and (3) How do specialized medical VLMs compare to generalist models in this context? Our results reveal that VLMs can effectively perform basic surgical perception tasks, such as object counting and localization, with performance levels comparable to general domain tasks. However, their performance deteriorates significantly when the tasks require medical knowledge. Notably, we find that specialized medical VLMs currently underperform compared to generalist models across both basic and advanced surgical tasks, suggesting that they are not yet optimized for the complexity of surgical environments. These findings highlight the need for further advancements to enable VLMs to handle the unique challenges posed by surgery. Overall, our work provides important insights for the development of next-generation endoscopic AI systems and identifies key areas for improvement in medical visual language models.