OVS Meets Continual Learning: Towards Sustainable Open-Vocabulary Segmentation

TL;DR

This paper introduces ConOVS, a continual learning approach for open-vocabulary segmentation that dynamically adapts to new data streams, outperforming existing methods in sequential data collection scenarios.

Contribution

The paper proposes ConOVS, a novel Mixture-of-Experts framework for continual learning in open-vocabulary segmentation, addressing limitations of existing approaches.

Findings

ConOVS outperforms existing methods across various datasets.

It effectively expands recognition capabilities with sequential data collection.

ConOVS demonstrates robustness in incremental and zero-shot scenarios.

Abstract

Open-Vocabulary Segmentation (OVS) aims to segment classes that are not present in the training dataset. However, most existing studies assume that the training data is fixed in advance, overlooking more practical scenarios where new datasets are continuously collected over time. To address this, we first analyze how existing OVS models perform under such conditions. In this context, we explore several approaches such as retraining, fine-tuning, and continual learning but find that each of them has clear limitations. To address these issues, we propose ConOVS, a novel continual learning method based on a Mixture-of-Experts framework. ConOVS dynamically combines expert decoders based on the probability that an input sample belongs to the distribution of each incremental dataset. Through extensive experiments, we show that ConOVS consistently outperforms existing methods across pre-training, incremental, and zero-shot test datasets, effectively expanding the recognition capabilities of OVS models when data is collected sequentially.