Learning Deep Sensorimotor Policies for Vision-based Autonomous Drone Racing

TL;DR

This paper presents a deep learning approach for vision-based autonomous drone racing, enabling drones to directly infer control commands from raw images without traditional state estimation or planning, achieving competitive performance.

Contribution

It introduces a contrastive learning-based feature extraction and a two-stage learning framework for training neural policies that operate directly on raw images for drone control.

Findings

Achieves racing performance comparable to state-based policies.

Robust against visual disturbances and distractors.

Eliminates need for handcrafted control components.

Abstract

Autonomous drones can operate in remote and unstructured environments, enabling various real-world applications. However, the lack of effective vision-based algorithms has been a stumbling block to achieving this goal. Existing systems often require hand-engineered components for state estimation, planning, and control. Such a sequential design involves laborious tuning, human heuristics, and compounding delays and errors. This paper tackles the vision-based autonomous-drone-racing problem by learning deep sensorimotor policies. We use contrastive learning to extract robust feature representations from the input images and leverage a two-stage learning-by-cheating framework for training a neural network policy. The resulting policy directly infers control commands with feature representations learned from raw images, forgoing the need for globally-consistent state estimation, trajectory planning, and handcrafted control design. Our experimental results indicate that our vision-based policy can achieve the same level of racing performance as the state-based policy while being robust against different visual disturbances and distractors. We believe this work serves as a stepping-stone toward developing intelligent vision-based autonomous systems that control the drone purely from image inputs, like human pilots.