Eye in the Sky: Drone-Based Object Tracking and 3D Localization

TL;DR

This paper presents a drone-based system that combines deep learning object detection, multi-object tracking, and 3D localization to effectively monitor and localize objects from aerial footage, addressing challenges like occlusion and fast motion.

Contribution

It introduces a novel integration of TrackletNet Tracker with 3D localization using Multi-View Stereo for drone applications, enhancing tracking and localization accuracy.

Findings

Reliable multi-object tracking on drone footage

Favorable 3D localization accuracy compared to state-of-the-art methods

Effective handling of occlusion and fast camera motion

Abstract

Drones, or general UAVs, equipped with a single camera have been widely deployed to a broad range of applications, such as aerial photography, fast goods delivery and most importantly, surveillance. Despite the great progress achieved in computer vision algorithms, these algorithms are not usually optimized for dealing with images or video sequences acquired by drones, due to various challenges such as occlusion, fast camera motion and pose variation. In this paper, a drone-based multi-object tracking and 3D localization scheme is proposed based on the deep learning based object detection. We first combine a multi-object tracking method called TrackletNet Tracker (TNT) which utilizes temporal and appearance information to track detected objects located on the ground for UAV applications. Then, we are also able to localize the tracked ground objects based on the group plane estimated from the Multi-View Stereo technique. The system deployed on the drone can not only detect and track the objects in a scene, but can also localize their 3D coordinates in meters with respect to the drone camera. The experiments have proved our tracker can reliably handle most of the detected objects captured by drones and achieve favorable 3D localization performance when compared with the state-of-the-art methods.