Learning from the Web: Language Drives Weakly-Supervised Incremental Learning for Semantic Segmentation

TL;DR

This paper introduces a novel web-image-based approach for weakly-supervised incremental semantic segmentation, leveraging web images and caption-driven rehearsal to learn new classes and preserve previous knowledge without manual annotations.

Contribution

It is the first to use solely web images for both learning new classes and retaining old ones in WILSS, employing a Fourier-based domain discriminator and caption-driven rehearsal.

Findings

Achieves state-of-the-art performance without manual annotations.

Effectively selects relevant web images for incremental learning.

Preserves learned classes while integrating new ones.

Abstract

Current weakly-supervised incremental learning for semantic segmentation (WILSS) approaches only consider replacing pixel-level annotations with image-level labels, while the training images are still from well-designed datasets. In this work, we argue that widely available web images can also be considered for the learning of new classes. To achieve this, firstly we introduce a strategy to select web images which are similar to previously seen examples in the latent space using a Fourier-based domain discriminator. Then, an effective caption-driven reharsal strategy is proposed to preserve previously learnt classes. To our knowledge, this is the first work to rely solely on web images for both the learning of new concepts and the preservation of the already learned ones in WILSS. Experimental results show that the proposed approach can reach state-of-the-art performances without using manually selected and annotated data in the incremental steps.