SurgGoal: Rethinking Surgical Planning Evaluation via Goal-Satisfiability

TL;DR

This paper introduces a goal-satisfiability based evaluation framework for surgical planning models, revealing limitations of existing metrics and emphasizing the importance of structural knowledge for improved performance.

Contribution

It proposes a new goal-oriented evaluation method for surgical planning, along with a multicentric benchmark and insights into model performance and failure modes.

Findings

Sequence similarity metrics often misjudge planning quality.

Structural knowledge enhances model performance.

Semantic guidance alone is unreliable, especially for larger models.

Abstract

Surgical planning integrates visual perception, long-horizon reasoning, and procedural knowledge, yet it remains unclear whether current evaluation protocols reliably assess vision-language models (VLMs) in safety-critical settings. Motivated by a goal-oriented view of surgical planning, we define planning correctness via phase-goal satisfiability, where plan validity is determined by expert-defined surgical rules. Based on this definition, we introduce a multicentric meta-evaluation benchmark with valid procedural variations and invalid plans containing order and content errors. Using this benchmark, we show that sequence similarity metrics systematically misjudge planning quality, penalizing valid plans while failing to identify invalid ones. We therefore adopt a rule-based goal-satisfiability metric as a high-precision meta-evaluation reference to assess Video-LLMs under progressively constrained settings, revealing failures due to perception errors and under-constrained reasoning. Structural knowledge consistently improves performance, whereas semantic guidance alone is unreliable and benefits larger models only when combined with structural constraints.