Is a Pure Transformer Effective for Separated and Online Multi-Object Tracking?

TL;DR

This paper introduces a novel Pure Transformer (PuTR) model for online multi-object tracking, effectively unifying short- and long-term association tasks with improved domain adaptation and practical efficiency.

Contribution

It proposes representing trajectory graphs as directed acyclic graphs and leveraging Transformer attention mechanisms, creating a new unified approach for online MOT.

Findings

PuTR outperforms existing methods on multiple datasets.

The approach demonstrates superior domain adaptation.

Efficient training and inference enable practical deployment.

Abstract

Recent advances in Multi-Object Tracking (MOT) have demonstrated significant success in short-term association within the separated tracking-by-detection online paradigm. However, long-term tracking remains challenging. While graph-based approaches address this by modeling trajectories as global graphs, these methods are unsuitable for real-time applications due to their non-online nature. In this paper, we review the concept of trajectory graphs and propose a novel perspective by representing them as directed acyclic graphs. This representation can be described using frame-ordered object sequences and binary adjacency matrices. We observe that this structure naturally aligns with Transformer attention mechanisms, enabling us to model the association problem using a classic Transformer architecture. Based on this insight, we introduce a concise Pure Transformer (PuTR) to validate the effectiveness of Transformer in unifying short- and long-term tracking for separated online MOT. Extensive experiments on four diverse datasets (SportsMOT, DanceTrack, MOT17, and MOT20) demonstrate that PuTR effectively establishes a solid baseline compared to existing foundational online methods while exhibiting superior domain adaptation capabilities. Furthermore, the separated nature enables efficient training and inference, making it suitable for practical applications. Implementation code and trained models are available at https://github.com/chongweiliu/PuTR .