Global Tracking Transformers

TL;DR

This paper introduces a global tracking transformer architecture that encodes object features across frames to produce trajectories without pairwise association, achieving high performance on multiple benchmarks.

Contribution

The novel global tracking transformer directly encodes object trajectories from all frames, eliminating the need for pairwise association and enabling seamless integration with large-vocabulary detectors.

Findings

Achieves 75.3 MOTA and 59.1 HOTA on MOT17

Outperforms existing pairwise association methods by 7.7 tracking mAP on TAO

Can be integrated with state-of-the-art detectors for diverse object tracking

Abstract

We present a novel transformer-based architecture for global multi-object tracking. Our network takes a short sequence of frames as input and produces global trajectories for all objects. The core component is a global tracking transformer that operates on objects from all frames in the sequence. The transformer encodes object features from all frames, and uses trajectory queries to group them into trajectories. The trajectory queries are object features from a single frame and naturally produce unique trajectories. Our global tracking transformer does not require intermediate pairwise grouping or combinatorial association, and can be jointly trained with an object detector. It achieves competitive performance on the popular MOT17 benchmark, with 75.3 MOTA and 59.1 HOTA. More importantly, our framework seamlessly integrates into state-of-the-art large-vocabulary detectors to track any objects. Experiments on the challenging TAO dataset show that our framework consistently improves upon baselines that are based on pairwise association, outperforming published works by a significant 7.7 tracking mAP. Code is available at https://github.com/xingyizhou/GTR.