AerialMind: Towards Referring Multi-Object Tracking in UAV Scenarios

TL;DR

AerialMind introduces a large-scale benchmark and a novel method for referring multi-object tracking in UAV scenarios, leveraging aerial perspectives and natural language instructions for improved scene understanding.

Contribution

The paper presents the first UAV-specific RMOT benchmark, a semi-automated annotation framework, and a new collaborative vision-language learning method called HawkEyeTrack.

Findings

The dataset is challenging and diverse.

HawkEyeTrack improves tracking accuracy.

The annotation framework reduces labeling costs.

Abstract

Referring Multi-Object Tracking (RMOT) aims to achieve precise object detection and tracking through natural language instructions, representing a fundamental capability for intelligent robotic systems. However, current RMOT research remains mostly confined to ground-level scenarios, which constrains their ability to capture broad-scale scene contexts and perform comprehensive tracking and path planning. In contrast, Unmanned Aerial Vehicles (UAVs) leverage their expansive aerial perspectives and superior maneuverability to enable wide-area surveillance. Moreover, UAVs have emerged as critical platforms for Embodied Intelligence, which has given rise to an unprecedented demand for intelligent aerial systems capable of natural language interaction. To this end, we introduce AerialMind, the first large-scale RMOT benchmark in UAV scenarios, which aims to bridge this research gap. To facilitate its construction, we develop an innovative semi-automated collaborative agent-based labeling assistant (COALA) framework that significantly reduces labor costs while maintaining annotation quality. Furthermore, we propose HawkEyeTrack (HETrack), a novel method that collaboratively enhances vision-language representation learning and improves the perception of UAV scenarios. Comprehensive experiments validated the challenging nature of our dataset and the effectiveness of our method.