Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation

TL;DR

This paper introduces a collaborative optimization framework for vision-text models like CLIP to improve open-vocabulary segmentation, balancing zero-shot capabilities with local region sensitivity through adaptive text enhancement and representation compensation.

Contribution

It proposes a novel collaborative vision-text optimization mechanism with content-dependent transfer and representation compensation, first in the OVS field, enhancing alignment and performance.

Findings

Achieves state-of-the-art results on multiple OVS benchmarks.

Outperforms previous methods by +0.5 to +3.4 mIoU on various datasets.

Attains competitive panoptic segmentation metrics on ADE20K.

Abstract

Pre-trained vision-language models, e.g. CLIP, have been increasingly used to address the challenging Open-Vocabulary Segmentation (OVS) task, benefiting from their well-aligned vision-text embedding space. Typical solutions involve either freezing CLIP during training to unilaterally maintain its zero-shot capability, or fine-tuning CLIP vision encoder to achieve perceptual sensitivity to local regions. However, few of them incorporate vision-text collaborative optimization. Based on this, we propose the Content-Dependent Transfer to adaptively enhance each text embedding by interacting with the input image, which presents a parameter-efficient way to optimize the text representation. Besides, we additionally introduce a Representation Compensation strategy, reviewing the original CLIP-V representation as compensation to maintain the zero-shot capability of CLIP. In this way, the vision and text representation of CLIP are optimized collaboratively, enhancing the alignment of the vision-text feature space. To the best of our knowledge, we are the first to establish the collaborative vision-text optimizing mechanism within the OVS field. Extensive experiments demonstrate our method achieves superior performance on popular OVS benchmarks. In open-vocabulary semantic segmentation, our method outperforms the previous state-of-the-art approaches by +0.5, +2.3, +3.4, +0.4 and +1.1 mIoU, respectively on A-847, A-150, PC-459, PC-59 and PAS-20. Furthermore, in a panoptic setting on ADE20K, we achieve the performance of 27.1 PQ, 73.5 SQ, and 32.9 RQ. Code will be available at https://github.com/jiaosiyu1999/MAFT-Plus.git .