UAV-Track VLA: Embodied Aerial Tracking via Vision-Language-Action Models

TL;DR

This paper introduces UAV-Track VLA, a novel vision-language-action model for UAV tracking, supported by a large-scale dataset and benchmark, demonstrating superior real-time performance and zero-shot generalization in urban scenarios.

Contribution

The paper presents a new VLA tracking model with temporal compression and dual-branch decoding, along with a comprehensive dataset and benchmark for UAV embodied tracking.

Findings

UAV-Track VLA achieves 61.76% success rate in long-distance pedestrian tracking.

The model reduces inference latency by 33.4% to 0.0571s.

Systematic experiments validate superior performance and zero-shot generalization.

Abstract

Embodied visual tracking is crucial for Unmanned Aerial Vehicles (UAVs) executing complex real-world tasks. In dynamic urban scenarios with complex semantic requirements, Vision-Language-Action (VLA) models show great promise due to their cross-modal fusion and continuous action generation capabilities. To benchmark multimodal tracking in such environments, we construct a dedicated evaluation benchmark and a large-scale dataset encompassing over 890K frames, 176 tasks, and 85 diverse objects. Furthermore, to address temporal feature redundancy and the lack of spatial geometric priors in existing VLA models, we propose an improved VLA tracking model, UAV-Track VLA. Built upon the $\pi_{0.5}$ architecture, our model introduces a temporal compression net to efficiently capture inter-frame dynamics. Additionally, a parallel dual-branch decoder comprising a spatial-aware auxiliary grounding head and a flow matching action expert is designed to decouple cross-modal features and generate fine-grained continuous actions. Systematic experiments in the CARLA simulator validate the superior end-to-end performance of our method. Notably, in challenging long-distance pedestrian tracking tasks, UAV-Track VLA achieves a 61.76\% success rate and 269.65 average tracking frames, significantly outperforming existing baselines. Furthermore, it demonstrates robust zero-shot generalization in unseen environments and reduces single-step inference latency by 33.4\% (to 0.0571s) compared to the original $\pi_{0.5}$, enabling highly efficient, real-time UAV control. Data samples and demonstration videos are available at: https://github.com/Hub-Tian/UAV-Track_VLA.