Linguistic Query-Guided Mask Generation for Referring Image Segmentation

TL;DR

This paper introduces LGFormer, a transformer-based framework that uses linguistic queries to generate image segmentation masks, improving cross-modal alignment and segmentation consistency for image-text pairs.

Contribution

The paper presents a novel end-to-end transformer model that dynamically generates prototypes based on linguistic queries for improved referring image segmentation.

Findings

Outperforms existing methods on benchmark datasets.

Achieves better cross-modal alignment and segmentation accuracy.

Demonstrates robustness across diverse image-text pairs.

Abstract

Referring image segmentation aims to segment the image region of interest according to the given language expression, which is a typical multi-modal task. Existing methods either adopt the pixel classification-based or the learnable query-based framework for mask generation, both of which are insufficient to deal with various text-image pairs with a fix number of parametric prototypes. In this work, we propose an end-to-end framework built on transformer to perform Linguistic query-Guided mask generation, dubbed LGFormer. It views the linguistic features as query to generate a specialized prototype for arbitrary input image-text pair, thus generating more consistent segmentation results. Moreover, we design several cross-modal interaction modules (\eg, vision-language bidirectional attention module, VLBA) in both encoder and decoder to achieve better cross-modal alignment.