Data-centric Design of Learning-based Surgical Gaze Perception Models in Multi-Task Simulation

TL;DR

This study investigates how different sources and modalities of gaze data influence the training of surgical perception models, proposing a scalable approach using passive gaze data for surgical training and modeling.

Contribution

It introduces a novel paired active-passive surgical gaze dataset and evaluates the effectiveness of passive gaze as a substitute for active gaze in training attention models.

Findings

Passive gaze can approximate active gaze with some degradation.

Models trained on passive gaze recover significant portions of active attention.

Novice passive labels are effective for higher-quality demonstrations.

Abstract

In robot-assisted minimally invasive surgery (RMIS), reduced haptic feedback and depth cues increase reliance on expert visual perception, motivating gaze-guided training and learning-based surgical perception models. However, operative expert gaze is costly to collect, and it remains unclear how the source of gaze supervision, both expertise level (intermediate vs. novice) and perceptual modality (active execution vs. passive viewing), shapes what attention models learn. We introduce a paired active-passive, multi-task surgical gaze dataset collected on the da Vinci SimNow simulator across four drills. Active gaze was recorded during task execution using a VR headset with eye tracking, and the corresponding videos were reused as stimuli to collect passive gaze from observers, enabling controlled same-video comparisons. We quantify skill- and modality-dependent differences in gaze organization and evaluate the substitutability of passive gaze for operative supervision using fixation density overlap analyses and single-frame saliency modeling. Across settings, MSI-Net produced stable, interpretable predictions, whereas SalGAN was unstable and often poorly aligned with human fixations. Models trained on passive gaze recovered a substantial portion of intermediate active attention, but with predictable degradation, and transfer was asymmetric between active and passive targets. Notably, novice passive labels approximated intermediate-passive targets with limited loss on higher-quality demonstrations, suggesting a practical path for scalable, crowd-sourced gaze supervision in surgical coaching and perception modeling.