Temporally Constrained Neural Networks (TCNN): A framework for semi-supervised video semantic segmentation

TL;DR

This paper introduces TCNN, a semi-supervised neural network framework that leverages temporal and spatial constraints to improve video semantic segmentation, especially in data-scarce, specialized fields like medicine.

Contribution

The paper presents a novel semi-supervised framework using autoencoders for temporal and spatial supervision in video segmentation, applicable across different models with minimal complexity.

Findings

Improved segmentation accuracy on surgical video datasets.

Effective use of low-dimensional representations for supervision.

Model-agnostic framework compatible with various architectures.

Abstract

A major obstacle to building models for effective semantic segmentation, and particularly video semantic segmentation, is a lack of large and well annotated datasets. This bottleneck is particularly prohibitive in highly specialized and regulated fields such as medicine and surgery, where video semantic segmentation could have important applications but data and expert annotations are scarce. In these settings, temporal clues and anatomical constraints could be leveraged during training to improve performance. Here, we present Temporally Constrained Neural Networks (TCNN), a semi-supervised framework used for video semantic segmentation of surgical videos. In this work, we show that autoencoder networks can be used to efficiently provide both spatial and temporal supervisory signals to train deep learning models. We test our method on a newly introduced video dataset of laparoscopic cholecystectomy procedures, Endoscapes, and an adaptation of a public dataset of cataract surgeries, CaDIS. We demonstrate that lower-dimensional representations of predicted masks can be leveraged to provide a consistent improvement on both sparsely labeled datasets with no additional computational cost at inference time. Further, the TCNN framework is model-agnostic and can be used in conjunction with other model design choices with minimal additional complexity.