SynRES: Towards Referring Expression Segmentation in the Wild via Synthetic Data

TL;DR

This paper introduces SynRES, a synthetic data generation pipeline that enhances referring expression segmentation models, enabling better generalization and performance on complex, real-world benchmarks like WildRES.

Contribution

The paper presents SynRES, a novel automated synthetic data generation method that improves RES model training and evaluation in diverse, complex scenarios.

Findings

Models trained with SynRES outperform previous methods on WildRES benchmarks.

SynRES improves gIoU by 2.0% on WildRES-ID and 3.8% on WildRES-DS.

WildRES reveals current RES models struggle with complex reasoning in real-world settings.

Abstract

Despite the advances in Referring Expression Segmentation (RES) benchmarks, their evaluation protocols remain constrained, primarily focusing on either single targets with short queries (containing minimal attributes) or multiple targets from distinctly different queries on a single domain. This limitation significantly hinders the assessment of more complex reasoning capabilities in RES models. We introduce WildRES, a novel benchmark that incorporates long queries with diverse attributes and non-distinctive queries for multiple targets. This benchmark spans diverse application domains, including autonomous driving environments and robotic manipulation scenarios, thus enabling more rigorous evaluation of complex reasoning capabilities in real-world settings. Our analysis reveals that current RES models demonstrate substantial performance deterioration when evaluated on WildRES. To address this challenge, we introduce SynRES, an automated pipeline generating densely paired compositional synthetic training data through three innovations: (1) a dense caption-driven synthesis for attribute-rich image-mask-expression triplets, (2) reliable semantic alignment mechanisms rectifying caption-pseudo mask inconsistencies via Image-Text Aligned Grouping, and (3) domain-aware augmentations incorporating mosaic composition and superclass replacement to emphasize generalization ability and distinguishing attributes over object categories. Experimental results demonstrate that models trained with SynRES achieve state-of-the-art performance, improving gIoU by 2.0% on WildRES-ID and 3.8% on WildRES-DS. Code and datasets are available at https://github.com/UTLLab/SynRES.