Finding NeMo: Negative-mined Mosaic Augmentation for Referring Image Segmentation

TL;DR

This paper introduces NeMo, a data augmentation technique for Referring Image Segmentation that creates challenging mosaic images with carefully selected negative samples to improve model discrimination and understanding.

Contribution

The paper proposes a novel Negative-mined Mosaic Augmentation method that enhances training data by incorporating carefully curated negative images, improving RIS model performance.

Findings

Consistent performance improvements across multiple datasets.

Effective in helping models distinguish subtle differences.

Highlights importance of difficulty level adjustment in data augmentation.

Abstract

Referring Image Segmentation is a comprehensive task to segment an object referred by a textual query from an image. In nature, the level of difficulty in this task is affected by the existence of similar objects and the complexity of the referring expression. Recent RIS models still show a significant performance gap between easy and hard scenarios. We pose that the bottleneck exists in the data, and propose a simple but powerful data augmentation method, Negative-mined Mosaic Augmentation (NeMo). This method augments a training image into a mosaic with three other negative images carefully curated by a pretrained multimodal alignment model, e.g., CLIP, to make the sample more challenging. We discover that it is critical to properly adjust the difficulty level, neither too ambiguous nor too trivial. The augmented training data encourages the RIS model to recognize subtle differences and relationships between similar visual entities and to concretely understand the whole expression to locate the right target better. Our approach shows consistent improvements on various datasets and models, verified by extensive experiments.