PencilNet: Zero-Shot Sim-to-Real Transfer Learning for Robust Gate Perception in Autonomous Drone Racing

TL;DR

PencilNet is a lightweight neural network that enables zero-shot sim-to-real transfer for robust gate detection in autonomous drone racing, effectively handling environmental variability without real-world training.

Contribution

The paper introduces PencilNet, a novel perception method that combines a pencil filter with a neural network for accurate, robust gate detection without real-world training data.

Findings

Effective zero-shot sim-to-real transfer without real-world data

Robust performance under varying illumination conditions

Successful detection in diverse challenging scenarios

Abstract

In autonomous and mobile robotics, one of the main challenges is the robust on-the-fly perception of the environment, which is often unknown and dynamic, like in autonomous drone racing. In this work, we propose a novel deep neural network-based perception method for racing gate detection -- PencilNet -- which relies on a lightweight neural network backbone on top of a pencil filter. This approach unifies predictions of the gates' 2D position, distance, and orientation in a single pose tuple. We show that our method is effective for zero-shot sim-to-real transfer learning that does not need any real-world training samples. Moreover, our framework is highly robust to illumination changes commonly seen under rapid flight compared to state-of-art methods. A thorough set of experiments demonstrates the effectiveness of this approach in multiple challenging scenarios, where the drone completes various tracks under different lighting conditions.