Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation

TL;DR

Mask-Adapter improves open-vocabulary segmentation by extracting richer semantic features and enforcing mask consistency, leading to significant performance gains across multiple benchmarks and models.

Contribution

The paper introduces Mask-Adapter, a novel method that enhances mask classification accuracy by extracting semantic activation maps and applying a mask consistency loss.

Findings

Significant performance improvements on zero-shot segmentation benchmarks.

Effective extension of Mask-Adapter to SAM model.

Robustness to varying predicted masks demonstrated.

Abstract

Recent open-vocabulary segmentation methods adopt mask generators to predict segmentation masks and leverage pre-trained vision-language models, e.g., CLIP, to classify these masks via mask pooling. Although these approaches show promising results, it is counterintuitive that accurate masks often fail to yield accurate classification results through pooling CLIP image embeddings within the mask regions. In this paper, we reveal the performance limitations of mask pooling and introduce Mask-Adapter, a simple yet effective method to address these challenges in open-vocabulary segmentation. Compared to directly using proposal masks, our proposed Mask-Adapter extracts semantic activation maps from proposal masks, providing richer contextual information and ensuring alignment between masks and CLIP. Additionally, we propose a mask consistency loss that encourages proposal masks with similar IoUs to obtain similar CLIP embeddings to enhance models' robustness to varying predicted masks. Mask-Adapter integrates seamlessly into open-vocabulary segmentation methods based on mask pooling in a plug-and-play manner, delivering more accurate classification results. Extensive experiments across several zero-shot benchmarks demonstrate significant performance gains for the proposed Mask-Adapter on several well-established methods. Notably, Mask-Adapter also extends effectively to SAM and achieves impressive results on several open-vocabulary segmentation datasets. Code and models are available at https://github.com/hustvl/MaskAdapter.