CFSSeg: Closed-Form Solution for Class-Incremental Semantic Segmentation of 2D Images and 3D Point Clouds

TL;DR

CFSSeg introduces a closed-form, exemplar-free method for class-incremental semantic segmentation of 2D images and 3D point clouds, significantly improving efficiency and avoiding catastrophic forgetting.

Contribution

It presents the first closed-form solution for continual semantic segmentation, eliminating the need for gradient-based training and data replay, and demonstrating superior performance on benchmarks.

Findings

Outperforms existing methods on Pascal VOC2012, S3DIS, and ScanNet datasets.

Reduces computational cost by avoiding iterative training.

Effectively mitigates catastrophic forgetting in incremental learning.

Abstract

2D images and 3D point clouds are foundational data types for multimedia applications, including real-time video analysis, augmented reality (AR), and 3D scene understanding. Class-incremental semantic segmentation (CSS) requires incrementally learning new semantic categories while retaining prior knowledge. Existing methods typically rely on computationally expensive training based on stochastic gradient descent, employing complex regularization or exemplar replay. However, stochastic gradient descent-based approaches inevitably update the model's weights for past knowledge, leading to catastrophic forgetting, a problem exacerbated by pixel/point-level granularity. To address these challenges, we propose CFSSeg, a novel exemplar-free approach that leverages a closed-form solution, offering a practical and theoretically grounded solution for continual semantic segmentation tasks. This eliminates the need for iterative gradient-based optimization and storage of past data, requiring only a single pass through new samples per step. It not only enhances computational efficiency but also provides a practical solution for dynamic, privacy-sensitive multimedia environments. Extensive experiments on 2D and 3D benchmark datasets such as Pascal VOC2012, S3DIS, and ScanNet demonstrate CFSSeg's superior performance.