Beyond One-to-One: Rethinking the Referring Image Segmentation

TL;DR

This paper introduces a Dual Multi-Modal Interaction network for referring image segmentation that handles complex expressions referring to multiple or no objects, supported by a new challenging dataset, Ref-ZOM.

Contribution

The paper proposes a novel DMMI network with dual decoders for improved segmentation and introduces the Ref-ZOM dataset for more realistic evaluation.

Findings

Achieves state-of-the-art performance on multiple datasets.

Performs well on various types of text inputs.

Demonstrates robustness in complex referring expressions.

Abstract

Referring image segmentation aims to segment the target object referred by a natural language expression. However, previous methods rely on the strong assumption that one sentence must describe one target in the image, which is often not the case in real-world applications. As a result, such methods fail when the expressions refer to either no objects or multiple objects. In this paper, we address this issue from two perspectives. First, we propose a Dual Multi-Modal Interaction (DMMI) Network, which contains two decoder branches and enables information flow in two directions. In the text-to-image decoder, text embedding is utilized to query the visual feature and localize the corresponding target. Meanwhile, the image-to-text decoder is implemented to reconstruct the erased entity-phrase conditioned on the visual feature. In this way, visual features are encouraged to contain the critical semantic information about target entity, which supports the accurate segmentation in the text-to-image decoder in turn. Secondly, we collect a new challenging but realistic dataset called Ref-ZOM, which includes image-text pairs under different settings. Extensive experiments demonstrate our method achieves state-of-the-art performance on different datasets, and the Ref-ZOM-trained model performs well on various types of text inputs. Codes and datasets are available at https://github.com/toggle1995/RIS-DMMI.