Drone Object Detection Using RGB/IR Fusion

TL;DR

This paper presents a novel approach for drone-based object detection by fusing RGB and IR images, utilizing synthetic data generation and an illumination-aware framework, achieving real-time performance on embedded hardware.

Contribution

The paper introduces strategies for synthetic IR image creation and an illumination-aware fusion framework for improved drone object detection.

Findings

Effective IR image synthesis using AIRSim and CycleGAN.

Fusion framework enhances detection in low-light conditions.

Real-time processing achieved on NVIDIA Jetson Xavier.

Abstract

Object detection using aerial drone imagery has received a great deal of attention in recent years. While visible light images are adequate for detecting objects in most scenarios, thermal cameras can extend the capabilities of object detection to night-time or occluded objects. As such, RGB and Infrared (IR) fusion methods for object detection are useful and important. One of the biggest challenges in applying deep learning methods to RGB/IR object detection is the lack of available training data for drone IR imagery, especially at night. In this paper, we develop several strategies for creating synthetic IR images using the AIRSim simulation engine and CycleGAN. Furthermore, we utilize an illumination-aware fusion framework to fuse RGB and IR images for object detection on the ground. We characterize and test our methods for both simulated and actual data. Our solution is implemented on an NVIDIA Jetson Xavier running on an actual drone, requiring about 28 milliseconds of processing per RGB/IR image pair.