MMNet: Multi-Mask Network for Referring Image Segmentation

TL;DR

This paper introduces MMNet, an end-to-end multi-mask network that generates multiple segmentation masks from natural language expressions, effectively handling language and object diversity for improved referring image segmentation.

Contribution

The paper proposes a novel multi-mask network that produces multiple segmentation masks and combines them to address uncertainty in referring image segmentation tasks.

Findings

Outperforms state-of-the-art on RefCOCO, RefCOCO+, and G-Ref datasets

Eliminates the need for post-processing in segmentation

Effectively reduces language-induced randomness

Abstract

Referring image segmentation aims to segment an object referred to by natural language expression from an image. However, this task is challenging due to the distinct data properties between text and image, and the randomness introduced by diverse objects and unrestricted language expression. Most of previous work focus on improving cross-modal feature fusion while not fully addressing the inherent uncertainty caused by diverse objects and unrestricted language. To tackle these problems, we propose an end-to-end Multi-Mask Network for referring image segmentation(MMNet). we first combine picture and language and then employ an attention mechanism to generate multiple queries that represent different aspects of the language expression. We then utilize these queries to produce a series of corresponding segmentation masks, assigning a score to each mask that reflects its importance. The final result is obtained through the weighted sum of all masks, which greatly reduces the randomness of the language expression. Our proposed framework demonstrates superior performance compared to state-of-the-art approaches on the two most commonly used datasets, RefCOCO, RefCOCO+ and G-Ref, without the need for any post-processing. This further validates the efficacy of our proposed framework.