kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies

TL;DR

kNN-CLIP introduces a training-free, memory-efficient method for continual open-vocabulary segmentation that avoids catastrophic forgetting and adapts to expanding vocabularies without retraining.

Contribution

It proposes a novel training-free approach, kNN-CLIP, that uses a database of embeddings to enable continual, large-vocabulary segmentation without retraining or high memory costs.

Findings

Achieves state-of-the-art results on large-vocabulary segmentation datasets.

Effectively adapts to expanding vocabularies without retraining.

Minimizes compute and memory costs compared to traditional methods.

Abstract

Continual segmentation has not yet tackled the challenge of improving open-vocabulary segmentation models with training data for accurate segmentation across large, continually expanding vocabularies. We discover that traditional continual training results in severe catastrophic forgetting, failing to outperform a zero-shot segmentation baseline. We introduce a novel training-free strategy, kNN-CLIP, which augments the model with a database of instance embeddings for semantic and panoptic segmentation that achieves zero forgetting. We demonstrate that kNN-CLIP can adapt to continually growing vocabularies without the need for retraining or large memory costs. kNN-CLIP enables open-vocabulary segmentation methods to expand their vocabularies on any domain with a single pass through the data, while only storing compact embeddings. This approach minimizes both compute and memory costs. kNN-CLIP achieves state-of-the-art performance across large-vocabulary semantic and panoptic segmentation datasets. We hope kNN-CLIP represents a significant step forward in enabling more efficient and adaptable continual segmentation, paving the way for advances in real-world large-vocabulary continual segmentation methods.