Towards Complex-query Referring Image Segmentation: A Novel Benchmark

TL;DR

This paper introduces RIS-CQ, a new benchmark dataset with complex language queries for Referring Image Segmentation, and proposes DuMoGa, a novel model that outperforms existing methods on this challenging benchmark.

Contribution

The paper creates RIS-CQ, a large-scale benchmark with complex queries, and develops DuMoGa, a dual-modality graph alignment model that advances RIS performance.

Findings

DuMoGa outperforms existing RIS methods on RIS-CQ.

RIS-CQ enables more realistic and challenging RIS research.

The benchmark facilitates evaluation of models with complex language queries.

Abstract

Referring Image Understanding (RIS) has been extensively studied over the past decade, leading to the development of advanced algorithms. However, there has been a lack of research investigating how existing algorithms should be benchmarked with complex language queries, which include more informative descriptions of surrounding objects and backgrounds (\eg \textit{"the black car."} vs. \textit{"the black car is parking on the road and beside the bus."}). Given the significant improvement in the semantic understanding capability of large pre-trained models, it is crucial to take a step further in RIS by incorporating complex language that resembles real-world applications. To close this gap, building upon the existing RefCOCO and Visual Genome datasets, we propose a new RIS benchmark with complex queries, namely \textbf{RIS-CQ}. The RIS-CQ dataset is of high quality and large scale, which challenges the existing RIS with enriched, specific and informative queries, and enables a more realistic scenario of RIS research. Besides, we present a nichetargeting method to better task the RIS-CQ, called dual-modality graph alignment model (\textbf{\textsc{DuMoGa}}), which outperforms a series of RIS methods.