A spatio-temporal network for video semantic segmentation in surgical videos

TL;DR

This paper introduces a novel spatio-temporal network architecture that enhances video semantic segmentation in surgical videos by improving temporal consistency and accuracy, which is crucial for clinical applications.

Contribution

The paper presents a new spatio-temporal decoder that can be added to existing segmentation models to improve temporal consistency in surgical video analysis.

Findings

Improved segmentation accuracy on CholecSeg8k and private datasets.

Enhanced temporal consistency across video frames.

Model adaptable to various segmentation encoders.

Abstract

Semantic segmentation in surgical videos has applications in intra-operative guidance, post-operative analytics and surgical education. Segmentation models need to provide accurate and consistent predictions since temporally inconsistent identification of anatomical structures can impair usability and hinder patient safety. Video information can alleviate these challenges leading to reliable models suitable for clinical use. We propose a novel architecture for modelling temporal relationships in videos. The proposed model includes a spatio-temporal decoder to enable video semantic segmentation by improving temporal consistency across frames. The encoder processes individual frames whilst the decoder processes a temporal batch of adjacent frames. The proposed decoder can be used on top of any segmentation encoder to improve temporal consistency. Model performance was evaluated on the CholecSeg8k dataset and a private dataset of robotic Partial Nephrectomy procedures. Segmentation performance was improved when the temporal decoder was applied across both datasets. The proposed model also displayed improvements in temporal consistency.