Self-supervised Learning via Cluster Distance Prediction for Operating Room Context Awareness

TL;DR

This paper introduces a novel 3D self-supervised learning method using ToF camera data to improve semantic segmentation and activity classification in operating rooms, reducing reliance on annotated data.

Contribution

The paper presents a new 3D self-supervised task based on predicting relative 3D distances in OR scenes, enhancing feature learning for clinical workflow understanding.

Findings

Significant performance improvements on two clinical datasets.

Enhanced effectiveness in low-annotation regimes.

Demonstrated utility of 3D spatial context learning.

Abstract

Semantic segmentation and activity classification are key components to creating intelligent surgical systems able to understand and assist clinical workflow. In the Operating Room, semantic segmentation is at the core of creating robots aware of clinical surroundings, whereas activity classification aims at understanding OR workflow at a higher level. State-of-the-art semantic segmentation and activity recognition approaches are fully supervised, which is not scalable. Self-supervision can decrease the amount of annotated data needed. We propose a new 3D self-supervised task for OR scene understanding utilizing OR scene images captured with ToF cameras. Contrary to other self-supervised approaches, where handcrafted pretext tasks are focused on 2D image features, our proposed task consists of predicting the relative 3D distance of image patches by exploiting the depth maps. Learning 3D spatial context generates discriminative features for our downstream tasks. Our approach is evaluated on two tasks and datasets containing multi-view data captured from clinical scenarios. We demonstrate a noteworthy improvement of performance on both tasks, specifically on low-regime data where utility of self-supervised learning is the highest.