A Multi-Drone Multi-View Dataset and Deep Learning Framework for Pedestrian Detection and Tracking

TL;DR

This paper introduces MATRIX, a challenging multi-drone dataset and a deep learning framework for pedestrian detection and tracking in complex urban environments with dynamic camera positions and occlusions.

Contribution

It provides a novel multi-drone dataset with synchronized footage and a deep learning framework that handles dynamic camera calibration, multi-view fusion, and robustness in complex scenarios.

Findings

Static camera methods perform well in simple environments but degrade in complex scenes.

The proposed framework maintains ~90% detection and tracking accuracy in challenging conditions.

Transfer learning significantly improves detection and tracking performance.

Abstract

Multi-drone surveillance systems offer enhanced coverage and robustness for pedestrian tracking, yet existing approaches struggle with dynamic camera positions and complex occlusions. This paper introduces MATRIX (Multi-Aerial TRacking In compleX environments), a comprehensive dataset featuring synchronized footage from eight drones with continuously changing positions, and a novel deep learning framework for multi-view detection and tracking. Unlike existing datasets that rely on static cameras or limited drone coverage, MATRIX provides a challenging scenario with 40 pedestrians and a significant architectural obstruction in an urban environment. Our framework addresses the unique challenges of dynamic drone-based surveillance through real-time camera calibration, feature-based image registration, and multi-view feature fusion in bird's-eye-view (BEV) representation. Experimental results demonstrate that while static camera methods maintain over 90\% detection and tracking precision and accuracy metrics in a simplified MATRIX environment without an obstruction, 10 pedestrians and a much smaller observational area, their performance significantly degrades in the complex environment. Our proposed approach maintains robust performance with $\sim$90\% detection and tracking accuracy, as well as successfully tracks $\sim$80\% of trajectories under challenging conditions. Transfer learning experiments reveal strong generalization capabilities, with the pretrained model achieving much higher detection and tracking accuracy performance compared to training the model from scratch. Additionally, systematic camera dropout experiments reveal graceful performance degradation, demonstrating practical robustness for real-world deployments where camera failures may occur. The MATRIX dataset and framework provide essential benchmarks for advancing dynamic multi-view surveillance systems.