AeroReformer: Aerial Referring Transformer for UAV-based Referring Image Segmentation

TL;DR

AeroReformer introduces a novel UAV-specific referring image segmentation framework utilizing a vision-language cross-attention module and rotation-aware multi-scale fusion, supported by newly created UAV datasets and automatic annotation pipeline.

Contribution

The paper presents AeroReformer, the first UAV-specific RIS model with a new dataset and an automatic annotation pipeline leveraging multimodal large language models.

Findings

AeroReformer outperforms existing methods on UAV-RIS datasets.

The automatic labeling pipeline effectively generates training data.

The proposed model establishes a new benchmark for UAV-based referring segmentation.

Abstract

As a novel and challenging task, referring segmentation combines computer vision and natural language processing to localize and segment objects based on textual descriptions. While referring image segmentation (RIS) has been extensively studied in natural images, little attention has been given to aerial imagery, particularly from unmanned aerial vehicles (UAVs). The unique challenges of UAV imagery, including complex spatial scales, occlusions, and varying object orientations, render existing RIS approaches ineffective. A key limitation has been the lack of UAV-specific datasets, as manually annotating pixel-level masks and generating textual descriptions is labour-intensive and time-consuming. To address this gap, we design an automatic labelling pipeline that leverages pre-existing UAV segmentation datasets and Multimodal Large Language Models (MLLM) for generating textual descriptions. Furthermore, we propose Aerial Referring Transformer (AeroReformer), a novel framework for UAV referring image segmentation (UAV-RIS), featuring a Vision-Language Cross-Attention Module (VLCAM) for effective cross-modal understanding and a Rotation-Aware Multi-Scale Fusion (RAMSF) decoder to enhance segmentation accuracy in aerial scenes. Extensive experiments on two newly developed datasets demonstrate the superiority of AeroReformer over existing methods, establishing a new benchmark for UAV-RIS. The datasets and code will be publicly available at: https://github.com/lironui/AeroReformer.