Recognizing Surgical Activities with Recurrent Neural Networks

TL;DR

This paper introduces the first application of recurrent neural networks to recognize both gestures and higher-level activities in robotic surgery, achieving state-of-the-art results in accuracy and edit distance.

Contribution

It demonstrates that recurrent neural networks can effectively recognize complex surgical activities from kinematic data, surpassing previous models in performance.

Findings

Achieved state-of-the-art accuracy for gesture recognition.

Improved maneuver recognition performance in terms of accuracy and edit distance.

Unified model with hyperparameters that generalize well across tasks.

Abstract

We apply recurrent neural networks to the task of recognizing surgical activities from robot kinematics. Prior work in this area focuses on recognizing short, low-level activities, or gestures, and has been based on variants of hidden Markov models and conditional random fields. In contrast, we work on recognizing both gestures and longer, higher-level activites, or maneuvers, and we model the mapping from kinematics to gestures/maneuvers with recurrent neural networks. To our knowledge, we are the first to apply recurrent neural networks to this task. Using a single model and a single set of hyperparameters, we match state-of-the-art performance for gesture recognition and advance state-of-the-art performance for maneuver recognition, in terms of both accuracy and edit distance. Code is available at https://github.com/rdipietro/miccai-2016-surgical-activity-rec .