AirSpatialBot: A Spatially-Aware Aerial Agent for Fine-Grained Vehicle Attribute Recognization and Retrieval

TL;DR

AirSpatialBot is a novel aerial agent that leverages a new spatially-aware dataset and training strategy to improve fine-grained vehicle attribute recognition and retrieval in drone imagery, addressing spatial understanding limitations of existing models.

Contribution

We introduce AirSpatial, a large spatially-aware remote sensing dataset with new tasks and a two-stage training strategy, enabling the development of a spatially-aware VLM-based aerial agent.

Findings

Validated the effectiveness of AirSpatialBot in vehicle attribute recognition and retrieval.

Identified spatial limitations of existing VLMs in remote sensing.

Provided new insights into spatial understanding in aerial vision-language models.

Abstract

Despite notable advancements in remote sensing vision-language models (VLMs), existing models often struggle with spatial understanding, limiting their effectiveness in real-world applications. To push the boundaries of VLMs in remote sensing, we specifically address vehicle imagery captured by drones and introduce a spatially-aware dataset AirSpatial, which comprises over 206K instructions and introduces two novel tasks: Spatial Grounding and Spatial Question Answering. It is also the first remote sensing grounding dataset to provide 3DBB. To effectively leverage existing image understanding of VLMs to spatial domains, we adopt a two-stage training strategy comprising Image Understanding Pre-training and Spatial Understanding Fine-tuning. Utilizing this trained spatially-aware VLM, we develop an aerial agent, AirSpatialBot, which is capable of fine-grained vehicle attribute recognition and retrieval. By dynamically integrating task planning, image understanding, spatial understanding, and task execution capabilities, AirSpatialBot adapts to diverse query requirements. Experimental results validate the effectiveness of our approach, revealing the spatial limitations of existing VLMs while providing valuable insights. The model, code, and datasets will be released at https://github.com/VisionXLab/AirSpatialBot