Bounded Future MS-TCN++ for surgical gesture recognition

TL;DR

This paper introduces a bounded future MS-TCN++ model for surgical gesture recognition, optimizing the trade-off between response delay and accuracy in real-time surgical video analysis.

Contribution

The study proposes a novel MS-TCN++ based approach that effectively manages the performance-delay trade-off for online surgical gesture recognition.

Findings

Limiting accessible future frames improves recognition performance.

Naive reduction of network depth is sub-optimal for small delays.

The proposed method outperforms naive approaches in delay-sensitive scenarios.

Abstract

In recent times there is a growing development of video based applications for surgical purposes. Part of these applications can work offline after the end of the procedure, other applications must react immediately. However, there are cases where the response should be done during the procedure but some delay is acceptable. In the literature, the online-offline performance gap is known. Our goal in this study was to learn the performance-delay trade-off and design an MS-TCN++-based algorithm that can utilize this trade-off. To this aim, we used our open surgery simulation data-set containing 96 videos of 24 participants that perform a suturing task on a variable tissue simulator. In this study, we used video data captured from the side view. The Networks were trained to identify the performed surgical gestures. The naive approach is to reduce the MS-TCN++ depth, as a result, the receptive field is reduced, and also the number of required future frames is also reduced. We showed that this method is sub-optimal, mainly in the small delay cases. The second method was to limit the accessible future in each temporal convolution. This way, we have flexibility in the network design and as a result, we achieve significantly better performance than in the naive approach.