Semantics-Aware Dynamic Localization and Refinement for Referring Image Segmentation

TL;DR

This paper introduces a simple, iterative approach for referring image segmentation that progressively refines multi-modal features using a dynamically updated query, improving segmentation quality over existing complex methods.

Contribution

The paper proposes a novel, versatile method that leverages a continuously updated query to enhance multi-modal feature learning for better segmentation results.

Findings

Outperforms state-of-the-art on RefCOCO, RefCOCO+, and G-Ref datasets.

More versatile and straightforward to integrate than existing methods.

Effectively recovers missing object parts and removes extraneous parts through iteration.

Abstract

Referring image segmentation segments an image from a language expression. With the aim of producing high-quality masks, existing methods often adopt iterative learning approaches that rely on RNNs or stacked attention layers to refine vision-language features. Despite their complexity, RNN-based methods are subject to specific encoder choices, while attention-based methods offer limited gains. In this work, we introduce a simple yet effective alternative for progressively learning discriminative multi-modal features. The core idea of our approach is to leverage a continuously updated query as the representation of the target object and at each iteration, strengthen multi-modal features strongly correlated to the query while weakening less related ones. As the query is initialized by language features and successively updated by object features, our algorithm gradually shifts from being localization-centric to segmentation-centric. This strategy enables the incremental recovery of missing object parts and/or removal of extraneous parts through iteration. Compared to its counterparts, our method is more versatile$\unicode{x2014}$it can be plugged into prior arts straightforwardly and consistently bring improvements. Experimental results on the challenging datasets of RefCOCO, RefCOCO+, and G-Ref demonstrate its advantage with respect to the state-of-the-art methods.