Multimodal Referring Segmentation: A Survey

TL;DR

This survey comprehensively reviews multimodal referring segmentation, covering problem definitions, datasets, methods across visual scenes, and performance benchmarks, highlighting recent advances and challenges in the field.

Contribution

It provides a unified overview of methods, datasets, and benchmarks in multimodal referring segmentation, emphasizing recent progress and future directions.

Findings

Significant performance improvements with CNNs, transformers, and large language models.

Unified meta architecture for referring segmentation across visual modalities.

Discussion of challenges and practical applications in real-world scenarios.

Abstract

Multimodal referring segmentation aims to segment target objects in visual scenes, such as images, videos, and 3D scenes, based on referring expressions in text or audio format. This task plays a crucial role in practical applications requiring accurate object perception based on user instructions. Over the past decade, it has gained significant attention in the multimodal community, driven by advances in convolutional neural networks, transformers, and large language models, all of which have substantially improved multimodal perception capabilities. This paper provides a comprehensive survey of multimodal referring segmentation. We begin by introducing this field's background, including problem definitions and commonly used datasets. Next, we summarize a unified meta architecture for referring segmentation and review representative methods across three primary visual scenes, including images, videos, and 3D scenes. We further discuss Generalized Referring Expression (GREx) methods to address the challenges of real-world complexity, along with related tasks and practical applications. Extensive performance comparisons on standard benchmarks are also provided. We continually track related works at https://github.com/henghuiding/Awesome-Multimodal-Referring-Segmentation.