GLSFormer: Gated - Long, Short Sequence Transformer for Step Recognition in Surgical Videos

TL;DR

GLSFormer introduces a gated-transformer model that jointly captures short and long-range spatio-temporal features for improved surgical step recognition in videos, outperforming existing methods on cataract surgery datasets.

Contribution

The paper presents a novel vision transformer with gated-temporal attention for joint spatio-temporal modeling in surgical videos, addressing limitations of prior separate or short-range methods.

Findings

Outperforms state-of-the-art methods on Cataract-101 and D99 datasets.

Effectively combines short-term and long-term features.

Demonstrates robustness across different surgical video datasets.

Abstract

Automated surgical step recognition is an important task that can significantly improve patient safety and decision-making during surgeries. Existing state-of-the-art methods for surgical step recognition either rely on separate, multi-stage modeling of spatial and temporal information or operate on short-range temporal resolution when learned jointly. However, the benefits of joint modeling of spatio-temporal features and long-range information are not taken in account. In this paper, we propose a vision transformer-based approach to jointly learn spatio-temporal features directly from sequence of frame-level patches. Our method incorporates a gated-temporal attention mechanism that intelligently combines short-term and long-term spatio-temporal feature representations. We extensively evaluate our approach on two cataract surgery video datasets, namely Cataract-101 and D99, and demonstrate superior performance compared to various state-of-the-art methods. These results validate the suitability of our proposed approach for automated surgical step recognition. Our code is released at: https://github.com/nisargshah1999/GLSFormer