Human Gaze Guided Attention for Surgical Activity Recognition

TL;DR

This paper introduces a novel approach that leverages human gaze data to guide spatio-temporal attention in surgical activity recognition, significantly improving accuracy on a public dataset.

Contribution

It is the first to incorporate human gaze as supervision for attention in surgical video activity recognition, enhancing model performance.

Findings

Achieved 85.4% accuracy on JIGSAWS Suturing task.

Demonstrated the effectiveness of gaze-guided attention over state-of-the-art models.

Validated through ablation studies the importance of gaze supervision.

Abstract

Modeling and automatically recognizing surgical activities are fundamental steps toward automation in surgery and play important roles in providing timely feedback to surgeons. Accurately recognizing surgical activities in video poses a challenging problem that requires an effective means of learning both spatial and temporal dynamics. Human gaze and visual saliency carry important information about visual attention and can be used to extract more relevant features that better reflect these spatial and temporal dynamics. In this study, we propose to use human gaze with a spatio-temporal attention mechanism for activity recognition in surgical videos. Our model consists of an I3D-based architecture, learns spatio-temporal features using 3D convolutions, as well as learns an attention map using human gaze as supervision. We evaluate our model on the Suturing task of JIGSAWS which is a publicly available surgical video understanding dataset. To our knowledge, we are the first to use human gaze for surgical activity recognition. Our results and ablation studies support the contribution of using human gaze to guide attention by outperforming state-of-the art models with an accuracy of 85.4%.