Continual Adaptation of Semantic Segmentation using Complementary 2D-3D Data Representations

TL;DR

This paper introduces a method for continual adaptation of semantic segmentation networks during deployment using 2D-3D data representations, improving accuracy without external supervision and retaining prior knowledge.

Contribution

It presents a novel continual learning approach that leverages volumetric 3D maps for self-supervised adaptation of segmentation networks in real-world environments.

Findings

Achieves a 9.9% average increase in segmentation accuracy.

Successfully adapts to real-world indoor scenes on multiple datasets.

Retains pre-trained knowledge while improving performance.

Abstract

Semantic segmentation networks are usually pre-trained once and not updated during deployment. As a consequence, misclassifications commonly occur if the distribution of the training data deviates from the one encountered during the robot's operation. We propose to mitigate this problem by adapting the neural network to the robot's environment during deployment, without any need for external supervision. Leveraging complementary data representations, we generate a supervision signal, by probabilistically accumulating consecutive 2D semantic predictions in a volumetric 3D map. We then train the network on renderings of the accumulated semantic map, effectively resolving ambiguities and enforcing multi-view consistency through the 3D representation. In contrast to scene adaptation methods, we aim to retain the previously-learned knowledge, and therefore employ a continual learning experience replay strategy to adapt the network. Through extensive experimental evaluation, we show successful adaptation to real-world indoor scenes both on the ScanNet dataset and on in-house data recorded with an RGB-D sensor. Our method increases the segmentation accuracy on average by 9.9% compared to the fixed pre-trained neural network, while retaining knowledge from the pre-training dataset.