Deep Learning based Multi-Modal Sensing for Tracking and State Extraction of Small Quadcopters

TL;DR

This paper introduces a multi-sensor system combining deep learning, thermal imaging, and lidar to detect, track, and localize small quadcopters with improved accuracy over existing methods.

Contribution

It presents a novel multi-sensor pipeline integrating RGB, thermal, and lidar data for UAV tracking and localization, enhancing current capabilities.

Findings

Favorable comparison to existing methods.

Effective UAV detection and tracking in various conditions.

Demonstrated accurate localization using combined sensors.

Abstract

This paper proposes a multi-sensor based approach to detect, track, and localize a quadcopter unmanned aerial vehicle (UAV). Specifically, a pipeline is developed to process monocular RGB and thermal video (captured from a fixed platform) to detect and track the UAV in our FoV. Subsequently, a 2D planar lidar is used to allow conversion of pixel data to actual distance measurements, and thereby enable localization of the UAV in global coordinates. The monocular data is processed through a deep learning-based object detection method that computes an initial bounding box for the UAV. The thermal data is processed through a thresholding and Kalman filter approach to detect and track the bounding box. Training and testing data are prepared by combining a set of original experiments conducted in a motion capture environment and publicly available UAV image data. The new pipeline compares favorably to existing methods and demonstrates promising tracking and localization capacity of sample experiments.