Dynamic Robot-Assisted Surgery with Hierarchical Class-Incremental Semantic Segmentation

TL;DR

This paper enhances class-incremental semantic segmentation for robotic surgery by developing TOPICS+ with hierarchical pseudo-labeling, addressing class imbalance, and creating new benchmarks for realistic surgical scene understanding.

Contribution

It introduces TOPICS+ with hierarchical pseudo-labeling and tailored taxonomies, along with new benchmarks for surgical scene segmentation, advancing continual learning in dynamic medical environments.

Findings

Improved segmentation accuracy on surgical benchmarks.

Effective handling of class imbalance with Dice loss.

Robust continual learning in evolving surgical scenes.

Abstract

Robot-assisted surgeries rely on accurate and real-time scene understanding to safely guide surgical instruments. However, segmentation models trained on static datasets face key limitations when deployed in these dynamic and evolving surgical environments. Class-incremental semantic segmentation (CISS) allows models to continually adapt to new classes while avoiding catastrophic forgetting of prior knowledge, without training on previous data. In this work, we build upon the recently introduced Taxonomy-Oriented Poincar\'e-regularized Incremental Class Segmentation (TOPICS) approach and propose an enhanced variant, termed TOPICS+, specifically tailored for robust segmentation of surgical scenes. Concretely, we incorporate the Dice loss into the hierarchical loss formulation to handle strong class imbalances, introduce hierarchical pseudo-labeling, and design tailored label taxonomies for robotic surgery environments. We also propose six novel CISS benchmarks designed for robotic surgery environments including multiple incremental steps and several semantic categories to emulate realistic class-incremental settings in surgical environments. In addition, we introduce a refined set of labels with more than 144 classes on the Syn-Mediverse synthetic dataset, hosted online as an evaluation benchmark. We make the code and trained models publicly available at http://topics.cs.uni-freiburg.de.