OnFly: Onboard Zero-Shot Aerial Vision-Language Navigation toward Safety and Efficiency

TL;DR

OnFly is a real-time onboard framework for aerial vision-language navigation that enhances safety and efficiency in UAVs by stabilizing decision-making, improving long-horizon monitoring, and refining target predictions.

Contribution

It introduces a dual-agent architecture, a hybrid memory system, and a semantic-geometric verifier for improved zero-shot aerial navigation.

Findings

Task success rate increased from 26.4% to 67.8%.

Validated feasibility through real-world onboard flights.

Achieved stable decision-making and reliable long-horizon monitoring.

Abstract

Aerial vision-language navigation (AVLN) enables UAVs to follow natural-language instructions in complex 3D environments. However, existing zero-shot AVLN methods often suffer from unstable single-stream Vision-Language Model decision-making, unreliable long-horizon progress monitoring, and a trade-off between safety and efficiency. We propose OnFly, a fully onboard, real-time framework for zero-shot AVLN. OnFly adopts a shared-perception dual-agent architecture that decouples high-frequency target generation from low-frequency progress monitoring, thereby stabilizing decision-making. It further employs a hybrid keyframe-recent-frame memory to preserve global trajectory context while maintaining KV-cache prefix stability, enabling reliable long-horizon monitoring with termination and recovery signals. In addition, a semantic-geometric verifier refines VLM-predicted targets for instruction consistency and geometric safety using VLM features and depth cues, while a receding-horizon planner generates optimized collision-free trajectories under geometric safety constraints, improving both safety and efficiency. In simulation, OnFly improves task success from 26.4% to 67.8%, compared with the strongest state-of-the-art baseline, while fully onboard real-world flights validate its feasibility for real-time deployment. The code will be released at https://github.com/Robotics-STAR-Lab/OnFly