Robotic Scene Segmentation with Memory Network for Runtime Surgical Context Inference

TL;DR

This paper introduces STCN, a memory network that improves real-time surgical scene segmentation and context inference by addressing class imbalance and ensuring temporal consistency, demonstrated on the JIGSAWS dataset.

Contribution

The paper presents a novel memory network architecture, STCN, that enhances surgical scene segmentation and context inference at runtime, overcoming limitations of existing methods.

Findings

STCN outperforms state-of-the-art segmentation methods on difficult classes.

Segmentation and context inference are feasible at runtime without performance loss.

Memory bank utilization improves temporal consistency of segmented masks.

Abstract

Surgical context inference has recently garnered significant attention in robot-assisted surgery as it can facilitate workflow analysis, skill assessment, and error detection. However, runtime context inference is challenging since it requires timely and accurate detection of the interactions among the tools and objects in the surgical scene based on the segmentation of video data. On the other hand, existing state-of-the-art video segmentation methods are often biased against infrequent classes and fail to provide temporal consistency for segmented masks. This can negatively impact the context inference and accurate detection of critical states. In this study, we propose a solution to these challenges using a Space Time Correspondence Network (STCN). STCN is a memory network that performs binary segmentation and minimizes the effects of class imbalance. The use of a memory bank in STCN allows for the utilization of past image and segmentation information, thereby ensuring consistency of the masks. Our experiments using the publicly available JIGSAWS dataset demonstrate that STCN achieves superior segmentation performance for objects that are difficult to segment, such as needle and thread, and improves context inference compared to the state-of-the-art. We also demonstrate that segmentation and context inference can be performed at runtime without compromising performance.