FAMNet: Joint Learning of Feature, Affinity and Multi-dimensional Assignment for Online Multiple Object Tracking

TL;DR

FAMNet introduces an end-to-end trainable model for multiple object tracking that jointly learns feature extraction, affinity estimation, and assignment, simplifying the pipeline and improving robustness.

Contribution

It proposes a fully differentiable network that integrates feature, affinity, and assignment learning for MOT, enabling joint optimization and better performance.

Findings

Achieves promising results on MOT2015, MOT2017, KITTI-Car, and UA-DETRAC benchmarks.

Outperforms several state-of-the-art methods in multiple metrics.

Effectively reduces false negatives and noisy detections through integrated tracking and target management.

Abstract

Data association-based multiple object tracking (MOT) involves multiple separated modules processed or optimized differently, which results in complex method design and requires non-trivial tuning of parameters. In this paper, we present an end-to-end model, named FAMNet, where Feature extraction, Affinity estimation and Multi-dimensional assignment are refined in a single network. All layers in FAMNet are designed differentiable thus can be optimized jointly to learn the discriminative features and higher-order affinity model for robust MOT, which is supervised by the loss directly from the assignment ground truth. We also integrate single object tracking technique and a dedicated target management scheme into the FAMNet-based tracking system to further recover false negatives and inhibit noisy target candidates generated by the external detector. The proposed method is evaluated on a diverse set of benchmarks including MOT2015, MOT2017, KITTI-Car and UA-DETRAC, and achieves promising performance on all of them in comparison with state-of-the-arts.