Revisiting the Evaluation Bias Introduced by Frame Sampling Strategies in Surgical Video Segmentation Using SAM2

TL;DR

This paper examines how different frame sampling strategies affect the evaluation of surgical video segmentation models, revealing biases that favor sparse sampling under certain conditions and emphasizing the importance of real-time, high-FPS evaluation for accurate assessment.

Contribution

It uncovers evaluation biases caused by inconsistent annotation protocols and demonstrates the superiority of high-FPS segmentation in real-time surgical video analysis.

Findings

Lower frame rates can falsely appear better due to smoothing effects.

High-FPS segmentation provides more stable and accurate results for dynamic objects.

Survey results favor high-FPS overlays for better human perception.

Abstract

Real-time video segmentation is a promising opportunity for AI-assisted surgery, offering intraoperative guidance by identifying tools and anatomical structures. Despite growing interest in surgical video segmentation, annotation protocols vary widely across datasets -- some provide dense, frame-by-frame labels, while others rely on sparse annotations sampled at low frame rates such as 1 FPS. In this study, we investigate how such inconsistencies in annotation density and frame rate sampling influence the evaluation of zero-shot segmentation models, using SAM2 as a case study for cholecystectomy procedures. Surprisingly, we find that under conventional sparse evaluation settings, lower frame rates can appear to outperform higher ones due to a smoothing effect that conceals temporal inconsistencies. However, when assessed under real-time streaming conditions, higher frame rates yield superior segmentation stability, particularly for dynamic objects like surgical graspers. To understand how these differences align with human perception, we conducted a survey among surgeons, nurses, and machine learning engineers and found that participants consistently preferred high-FPS segmentation overlays, reinforcing the importance of evaluating every frame in real-time applications rather than relying on sparse sampling strategies. Our findings highlight the risk of evaluation bias that is introduced by inconsistent dataset protocols and bring attention to the need for temporally fair benchmarking in surgical video AI.