Effects of Architectures on Continual Semantic Segmentation

TL;DR

This paper investigates how different neural network architectures, including CNNs, Transformers, and hybrids, influence catastrophic forgetting in continual semantic segmentation, highlighting architecture choice as a key factor.

Contribution

It provides a comprehensive comparison of architectures and normalization layers, revealing their impact on stability and plasticity in continual learning for semantic segmentation.

Findings

Transformers are more stable than CNNs in continual learning.

Hybrid architectures balance plasticity and stability effectively.

Continual Normalization improves adaptability and stability.

Abstract

Research in the field of Continual Semantic Segmentation is mainly investigating novel learning algorithms to overcome catastrophic forgetting of neural networks. Most recent publications have focused on improving learning algorithms without distinguishing effects caused by the choice of neural architecture.Therefore, we study how the choice of neural network architecture affects catastrophic forgetting in class- and domain-incremental semantic segmentation. Specifically, we compare the well-researched CNNs to recently proposed Transformers and Hybrid architectures, as well as the impact of the choice of novel normalization layers and different decoder heads. We find that traditional CNNs like ResNet have high plasticity but low stability, while transformer architectures are much more stable. When the inductive biases of CNN architectures are combined with transformers in hybrid architectures, it leads to higher plasticity and stability. The stability of these models can be explained by their ability to learn general features that are robust against distribution shifts. Experiments with different normalization layers show that Continual Normalization achieves the best trade-off in terms of adaptability and stability of the model. In the class-incremental setting, the choice of the normalization layer has much less impact. Our experiments suggest that the right choice of architecture can significantly reduce forgetting even with naive fine-tuning and confirm that for real-world applications, the architecture is an important factor in designing a continual learning model.