Deployable Vision-driven UAV River Navigation via Human-in-the-loop Preference Alignment

TL;DR

This paper presents SPAR-H, a human-in-the-loop learning method that efficiently adapts vision-driven UAV policies for river navigation by aligning with human preferences, improving safety and performance in real-world deployment.

Contribution

Introduction of SPAR-H, a novel hybrid preference alignment method that combines direct preference optimization with reward estimation for UAV river navigation.

Findings

SPAR-H achieves higher episodic rewards and lower variance than other methods.

The learned reward model aligns well with human preferences.

SPAR-H demonstrates real-world feasibility for UAV river following.

Abstract

Rivers are critical corridors for environmental monitoring and disaster response, where Unmanned Aerial Vehicles (UAVs) guided by vision-driven policies can provide fast, low-cost coverage. However, deployment exposes simulation-trained policies with distribution shift and safety risks and requires efficient adaptation from limited human interventions. We study human-in-the-loop (HITL) learning with a conservative overseer who vetoes unsafe or inefficient actions and provides statewise preferences by comparing the agent's proposal with a corrective override. We introduce Statewise Hybrid Preference Alignment for Robotics (SPAR-H), which fuses direct preference optimization on policy logits with a reward-based pathway that trains an immediate-reward estimator from the same preferences and updates the policy using a trust-region surrogate. With five HITL rollouts collected from a fixed novice policy, SPAR-H achieves the highest final episodic reward and the lowest variance across initial conditions among tested methods. The learned reward model aligns with human-preferred actions and elevates nearby non-intervened choices, supporting stable propagation of improvements. We benchmark SPAR-H against imitation learning (IL), direct preference variants, and evaluative reinforcement learning (RL) in the HITL setting, and demonstrate real-world feasibility of continual preference alignment for UAV river following. Overall, dual statewise preferences empirically provide a practical route to data-efficient online adaptation in riverine navigation.