Vision Transformers for End-to-End Vision-Based Quadrotor Obstacle Avoidance

TL;DR

This paper explores the use of vision transformers for end-to-end high-speed obstacle avoidance in quadrotors, demonstrating superior performance over traditional architectures in simulation and hardware tests.

Contribution

It introduces the first application of vision transformers for end-to-end quadrotor control, showing improved accuracy and generalization at high speeds compared to other neural network architectures.

Findings

Vision transformers outperform CNNs and U-Nets at high speeds.

Recurrent models further enhance performance and reduce energy consumption.

Successful real-world implementation up to 7 m/s.

Abstract

We demonstrate the capabilities of an attention-based end-to-end approach for high-speed vision-based quadrotor obstacle avoidance in dense, cluttered environments, with comparison to various state-of-the-art learning architectures. Quadrotor unmanned aerial vehicles (UAVs) have tremendous maneuverability when flown fast; however, as flight speed increases, traditional model-based approaches to navigation via independent perception, mapping, planning, and control modules breaks down due to increased sensor noise, compounding errors, and increased processing latency. Thus, learning-based, end-to-end vision-to-control networks have shown to have great potential for online control of these fast robots through cluttered environments. We train and compare convolutional, U-Net, and recurrent architectures against vision transformer (ViT) models for depth image-to-control in high-fidelity simulation, observing that ViT models are more effective than others as quadrotor speeds increase and in generalization to unseen environments, while the addition of recurrence further improves performance while reducing quadrotor energy cost across all tested flight speeds. We assess performance at speeds of up to 7m/s in simulation and hardware. To the best of our knowledge, this is the first work to utilize vision transformers for end-to-end vision-based quadrotor control.