Deep Learning-Based Multi-Object Tracking: A Comprehensive Survey from Foundations to State-of-the-Art

TL;DR

This comprehensive survey reviews deep learning-based multi-object tracking methods, categorizing approaches, analyzing recent advancements like ByteTrack and MOTR, and evaluating their performance across various benchmarks and scenarios.

Contribution

It systematically categorizes deep learning-based MOT methods, compares end-to-end and tracking-by-detection approaches, and provides performance insights across multiple datasets.

Findings

Heuristic methods excel in densely populated, linear motion scenarios.

Deep learning association methods perform well with complex motion patterns.

Recent methods like ByteTrack and MOTR have advanced the state-of-the-art.

Abstract

Multi-object tracking (MOT) is a core task in computer vision that involves detecting objects in video frames and associating them across time. The rise of deep learning has significantly advanced MOT, particularly within the tracking-by-detection paradigm, which remains the dominant approach. Advancements in modern deep learning-based methods accelerated in 2022 with the introduction of ByteTrack for tracking-by-detection and MOTR for end-to-end tracking. Our survey provides an in-depth analysis of deep learning-based MOT methods, systematically categorizing tracking-by-detection approaches into five groups: joint detection and embedding, heuristic-based, motion-based, affinity learning, and offline methods. In addition, we examine end-to-end tracking methods and compare them with existing alternative approaches. We evaluate the performance of recent trackers across multiple benchmarks and specifically assess their generality by comparing results across different domains. Our findings indicate that heuristic-based methods achieve state-of-the-art results on densely populated datasets with linear object motion, while deep learning-based association methods, in both tracking-by-detection and end-to-end approaches, excel in scenarios with complex motion patterns.