CLIMB-3D: Continual Learning for Imbalanced 3D Instance Segmentation

TL;DR

CLIMB-3D introduces a novel framework for continual learning in 3D instance segmentation that effectively handles class imbalance and emerging classes, achieving state-of-the-art results on challenging datasets.

Contribution

The paper proposes a unified framework combining exemplar replay, a pseudo-label generator, and class-balanced re-weighting to improve continual 3D instance segmentation under class imbalance.

Findings

Achieves up to 16.76% mAP improvement on ScanNet200.

Demonstrates strong generalization to rare classes.

Validates effectiveness on both 3D instance and semantic segmentation.

Abstract

While 3D instance segmentation (3DIS) has advanced significantly, most existing methods assume that all object classes are known in advance and uniformly distributed. However, this assumption is unrealistic in dynamic, real-world environments where new classes emerge gradually and exhibit natural imbalance. Although some approaches address the emergence of new classes, they often overlook class imbalance, which leads to suboptimal performance, particularly on rare categories. To tackle this, we propose \ourmethodbf, a unified framework for \textbf{CL}ass-incremental \textbf{Imb}alance-aware \textbf{3D}IS. Building upon established exemplar replay (ER) strategies, we show that ER alone is insufficient to achieve robust performance under memory constraints. To mitigate this, we introduce a novel pseudo-label generator (PLG) that extends supervision to previously learned categories by leveraging predictions from a frozen model trained on prior tasks. Despite its promise, PLG tends to be biased towards frequent classes. Therefore, we propose a class-balanced re-weighting (CBR) scheme that estimates object frequencies from pseudo-labels and dynamically adjusts training bias, without requiring access to past data. We design and evaluate three incremental scenarios for 3DIS on the challenging ScanNet200 dataset and additionally validate our method for semantic segmentation on ScanNetV2. Our approach achieves state-of-the-art results, surpassing prior work by up to 16.76\% mAP for instance segmentation and approximately 30\% mIoU for semantic segmentation, demonstrating strong generalisation across both frequent and rare classes. Code is available at: https://github.com/vgthengane/CLIMB3D