Bring Adaptive Binding Prototypes to Generalized Referring Expression Segmentation

TL;DR

This paper introduces a novel adaptive binding prototype model for generalized referring expression segmentation, significantly improving performance over existing methods by enhancing query-instance matching flexibility.

Contribution

The paper proposes the Model with Adaptive Binding Prototypes (MABP), which adaptively binds queries to object features, addressing limitations of prior RES methods in complex GRES scenarios.

Findings

MABP outperforms state-of-the-art methods on gRefCOCO dataset.

MABP surpasses existing methods on RefCOCO+ and G-Ref datasets.

Achieves competitive results on RefCOCO.

Abstract

Referring Expression Segmentation (RES) has attracted rising attention, aiming to identify and segment objects based on natural language expressions. While substantial progress has been made in RES, the emergence of Generalized Referring Expression Segmentation (GRES) introduces new challenges by allowing expressions to describe multiple objects or lack specific object references. Existing RES methods, usually rely on sophisticated encoder-decoder and feature fusion modules, and are difficult to generate class prototypes that match each instance individually when confronted with the complex referent and binary labels of GRES. In this paper, reevaluating the differences between RES and GRES, we propose a novel Model with Adaptive Binding Prototypes (MABP) that adaptively binds queries to object features in the corresponding region. It enables different query vectors to match instances of different categories or different parts of the same instance, significantly expanding the decoder's flexibility, dispersing global pressure across all queries, and easing the demands on the encoder. Experimental results demonstrate that MABP significantly outperforms state-of-the-art methods in all three splits on gRefCOCO dataset. Meanwhile, MABP also surpasses state-of-the-art methods on RefCOCO+ and G-Ref datasets, and achieves very competitive results on RefCOCO. Code is available at https://github.com/buptLwz/MABP