AR-MOT: Autoregressive Multi-object Tracking

TL;DR

AR-MOT introduces an autoregressive, sequence generation approach to multi-object tracking within a large language model framework, enabling flexible, task-agnostic, and extensible tracking without task-specific architectures.

Contribution

It proposes a novel autoregressive paradigm for MOT that eliminates fixed output heads, incorporates a region-aware alignment, and supports long-term tracking through sequence-based modeling.

Findings

Achieves performance comparable to state-of-the-art on MOT17 and DanceTrack.

Demonstrates flexible integration of new modalities and instructions.

Validates the effectiveness of the autoregressive approach for general MOT tasks.

Abstract

As multi-object tracking (MOT) tasks continue to evolve toward more general and multi-modal scenarios, the rigid and task-specific architectures of existing MOT methods increasingly hinder their applicability across diverse tasks and limit flexibility in adapting to new tracking formulations. Most approaches rely on fixed output heads and bespoke tracking pipelines, making them difficult to extend to more complex or instruction-driven tasks. To address these limitations, we propose AR-MOT, a novel autoregressive paradigm that formulates MOT as a sequence generation task within a large language model (LLM) framework. This design enables the model to output structured results through flexible sequence construction, without requiring any task-specific heads. To enhance region-level visual perception, we introduce an Object Tokenizer based on a pretrained detector. To mitigate the misalignment between global and regional features, we propose a Region-Aware Alignment (RAA) module, and to support long-term tracking, we design a Temporal Memory Fusion (TMF) module that caches historical object tokens. AR-MOT offers strong potential for extensibility, as new modalities or instructions can be integrated by simply modifying the output sequence format without altering the model architecture. Extensive experiments on MOT17 and DanceTrack validate the feasibility of our approach, achieving performance comparable to state-of-the-art methods while laying the foundation for more general and flexible MOT systems.