AdaptFly: Prompt-Guided Adaptation of Foundation Models for Low-Altitude UAV Networks

TL;DR

AdaptFly introduces a prompt-guided, weight-free test-time adaptation framework for UAV semantic segmentation, enhancing robustness and accuracy under challenging environmental conditions without heavy resource demands.

Contribution

It proposes a novel prompt-based adaptation method tailored for resource-constrained UAVs, enabling effective fleet-wide model improvement without weight updates.

Findings

Significant accuracy improvements on UAVid and VDD benchmarks.

Effective adaptation under diverse weather and lighting conditions.

Low bandwidth communication for fleet collaboration.

Abstract

Low-altitude Unmanned Aerial Vehicle (UAV) networks rely on robust semantic segmentation as a foundational enabler for distributed sensing-communication-control co-design across heterogeneous agents within the network. However, segmentation foundation models deteriorate quickly under weather, lighting, and viewpoint drift. Resource-limited UAVs cannot run gradient-based test-time adaptation, while resource-massive UAVs adapt independently, wasting shared experience. To address these challenges, we propose AdaptFly, a prompt-guided test-time adaptation framework that adjusts segmentation models without weight updates. AdaptFly features two complementary adaptation modes. For resource-limited UAVs, it employs lightweight token-prompt retrieval from a shared global memory. For resource-massive UAVs, it uses gradient-free sparse visual prompt optimization via Covariance Matrix Adaptation Evolution Strategy. An activation-statistic detector triggers adaptation, while cross-UAV knowledge pool consolidates prompt knowledge and enables fleet-wide collaboration with negligible bandwidth overhead. Extensive experiments on UAVid and VDD benchmarks, along with real-world UAV deployments under diverse weather conditions, demonstrate that AdaptFly significantly improves segmentation accuracy and robustness over static models and state-of-the-art TTA baselines. The results highlight a practical path to resilient, communication-efficient perception in the emerging low-altitude economy.