SelfMOTR: Revisiting MOTR with Self-Generating Detection Priors

TL;DR

SelfMOTR introduces a detector-free multi-object tracking method that leverages self-generated detection priors within an end-to-end transformer framework, improving tracking performance without external detectors.

Contribution

It proposes a novel detector-free approach that decouples proposal discovery from association using internal detection priors, enhancing end-to-end transformer tracking.

Findings

Achieves 69.2 HOTA on DanceTrack.

Leads with 71.1 HOTA on Bird Flock Tracking.

Shows that joint detection-association decoding retains hidden detection capacity.

Abstract

End-to-end transformer architectures have driven significant progress in multi-object tracking by unifying detection and association into a single, heuristic-free framework. Despite these benefits, poor detection performance and the inherent conflict between detection and association in a joint architecture remain critical concerns. Recent approaches aim to mitigate these issues by employing advanced denoising or label assignment strategies, or by incorporating detection priors from external object detectors. In this paper, we propose SelfMOTR, a simple yet highly effective detector-free alternative that decouples proposal discovery from association using self-generated internal detection priors. Through extensive analysis and ablation studies, we show that end-to-end transformer trackers with joint detection-association decoding retain substantial hidden detection capacity, and we provide a practical detector-free mechanism for leveraging it. To shed light on these joint decoding dynamics, we draw inspiration from attention sink analyses in large language models, leveraging Track Attention Mass to show that standard generic queries exhibit unbalanced attention, frequently struggling to weigh track context against novel object discovery. SelfMOTR achieves highly competitive performance in complex, dynamic environments, yielding 69.2 HOTA on DanceTrack and leading with 71.1 HOTA on the Bird Flock Tracking (BFT) dataset. Project page: https://medem23.github.io/SM