CRIS: CLIP-Driven Referring Image Segmentation

TL;DR

CRIS leverages CLIP's multi-modal knowledge through vision-language decoding and contrastive learning to improve referring image segmentation, achieving state-of-the-art results without post-processing.

Contribution

This paper introduces a novel end-to-end framework that effectively transfers multi-modal knowledge for segmentation using CLIP, vision-language decoding, and contrastive learning.

Findings

Significantly outperforms existing methods on benchmark datasets.

No post-processing needed for high performance.

Demonstrates effective multi-modal knowledge transfer.

Abstract

Referring image segmentation aims to segment a referent via a natural linguistic expression.Due to the distinct data properties between text and image, it is challenging for a network to well align text and pixel-level features. Existing approaches use pretrained models to facilitate learning, yet separately transfer the language/vision knowledge from pretrained models, ignoring the multi-modal corresponding information. Inspired by the recent advance in Contrastive Language-Image Pretraining (CLIP), in this paper, we propose an end-to-end CLIP-Driven Referring Image Segmentation framework (CRIS). To transfer the multi-modal knowledge effectively, CRIS resorts to vision-language decoding and contrastive learning for achieving the text-to-pixel alignment. More specifically, we design a vision-language decoder to propagate fine-grained semantic information from textual representations to each pixel-level activation, which promotes consistency between the two modalities. In addition, we present text-to-pixel contrastive learning to explicitly enforce the text feature similar to the related pixel-level features and dissimilar to the irrelevances. The experimental results on three benchmark datasets demonstrate that our proposed framework significantly outperforms the state-of-the-art performance without any post-processing. The code will be released.