Learning Association via Track-Detection Matching for Multi-Object Tracking

TL;DR

This paper introduces TDLP, a data-driven, modular, and efficient link prediction method for multi-object tracking that outperforms existing heuristics and end-to-end approaches across multiple benchmarks.

Contribution

The paper presents a novel link prediction approach for data association in multi-object tracking, combining efficiency with learned association without handcrafted rules.

Findings

TDLP surpasses state-of-the-art methods on multiple benchmarks.

Link prediction outperforms metric learning for heterogeneous features.

The approach is modular and computationally efficient.

Abstract

Multi-object tracking aims to maintain object identities over time by associating detections across video frames. Two dominant paradigms exist in literature: tracking-by-detection methods, which are computationally efficient but rely on handcrafted association heuristics, and end-to-end approaches, which learn association from data at the cost of higher computational complexity. We propose Track-Detection Link Prediction (TDLP), a tracking-by-detection method that performs per-frame association via link prediction between tracks and detections, i.e., by predicting the correct continuation of each track at every frame. TDLP is architecturally designed primarily for geometric features such as bounding boxes, while optionally incorporating additional cues, including pose and appearance. Unlike heuristic-based methods, TDLP learns association directly from data without handcrafted rules, while remaining modular and computationally efficient compared to end-to-end trackers. Extensive experiments on multiple benchmarks demonstrate that TDLP consistently surpasses state-of-the-art performance across both tracking-by-detection and end-to-end methods. Finally, we provide a detailed analysis comparing link prediction with metric learning-based association and show that link prediction is more effective, particularly when handling heterogeneous features such as detection bounding boxes. Our code is available at \href{https://github.com/Robotmurlock/TDLP}{https://github.com/Robotmurlock/TDLP}.