Vision-Based End-to-End Learning for UAV Traversal of Irregular Gaps via Differentiable Simulation

TL;DR

This paper introduces a vision-based end-to-end method for autonomous drones to traverse complex, irregular gaps using differentiable simulation and auxiliary modules, demonstrating robustness in simulation and real-world tests.

Contribution

The work presents a novel fully vision-based framework that directly maps depth images to control commands for gap traversal, incorporating differentiable simulation and auxiliary predictors.

Findings

Effective in simulation and real-world environments

Generalizes to unseen complex gaps

Enhances safety with auxiliary prediction modules

Abstract

-Navigation through narrow and irregular gaps is an essential skill in autonomous drones for applications such as inspection, search-and-rescue, and disaster response. However, traditional planning and control methods rely on explicit gap extraction and measurement, while recent end-to-end approaches often assume regularly shaped gaps, leading to poor generalization and limited practicality. In this work, we present a fully vision-based, end-to-end framework that maps depth images directly to control commands, enabling drones to traverse complex gaps within unseen environments. Operating in the Special Euclidean group SE(3), where position and orientation are tightly coupled, the framework leverages differentiable simulation, a Stop-Gradient operator, and a Bimodal Initialization Distribution to achieve stable traversal through consecutive gaps. Two auxiliary prediction modules-a gap-crossing success classifier and a traversability predictor-further enhance continuous navigation and safety. Extensive simulation and real-world experiments demonstrate the approach's effectiveness, generalization capability, and practical robustness.