High-throughput Visual Nano-drone to Nano-drone Relative Localization using Onboard Fully Convolutional Networks

TL;DR

This paper introduces a lightweight, real-time vision-based neural network system for relative localization of nano-drones, enabling efficient swarm operations with limited onboard resources.

Contribution

It presents a novel fully convolutional neural network that runs onboard nano-drones at 39Hz, improving localization accuracy and energy efficiency in resource-constrained environments.

Findings

Achieves 39Hz processing at 101mW onboard

Improves R-squared for pose estimation from 32% to 47% horizontally

Reduces tracking error by 37% in field tests

Abstract

Relative drone-to-drone localization is a fundamental building block for any swarm operations. We address this task in the context of miniaturized nano-drones, i.e., 10cm in diameter, which show an ever-growing interest due to novel use cases enabled by their reduced form factor. The price for their versatility comes with limited onboard resources, i.e., sensors, processing units, and memory, which limits the complexity of the onboard algorithms. A traditional solution to overcome these limitations is represented by lightweight deep learning models directly deployed aboard nano-drones. This work tackles the challenging relative pose estimation between nano-drones using only a gray-scale low-resolution camera and an ultra-low-power System-on-Chip (SoC) hosted onboard. We present a vertically integrated system based on a novel vision-based fully convolutional neural network (FCNN), which runs at 39Hz within 101mW onboard a Crazyflie nano-drone extended with the GWT GAP8 SoC. We compare our FCNN against three State-of-the-Art (SoA) systems. Considering the best-performing SoA approach, our model results in an R-squared improvement from 32 to 47% on the horizontal image coordinate and from 18 to 55% on the vertical image coordinate, on a real-world dataset of 30k images. Finally, our in-field tests show a reduction of the average tracking error of 37% compared to a previous SoA work and an endurance performance up to the entire battery lifetime of 4 minutes.