Offline-Poly: A Polyhedral Framework For Offline 3D Multi-Object Tracking

TL;DR

Offline-Poly introduces a flexible, tracking-centric framework for offline 3D multi-object tracking that leverages global optimization and future observability to improve accuracy and robustness beyond existing online-based methods.

Contribution

The paper proposes Offline-Poly, a novel offline 3D MOT framework based on a standardized Tracking-by-Tracking paradigm that decouples the tracker from specific detectors and enhances tracklet refinement.

Findings

Achieves state-of-the-art performance on nuScenes with 77.6% AMOTA.

Attains leading results on KITTI with 83.00% HOTA.

Demonstrates high flexibility and generalizability across different datasets.

Abstract

Offline 3D multi-object tracking (MOT) is a critical component of the 4D auto-labeling (4DAL) process. It enhances pseudo-labels generated by high-performance detectors through the incorporation of temporal context. However, existing offline 3D MOT approaches are direct extensions of online frameworks and fail to fully exploit the advantages of offline setting. Moreover, these methods often depend on fixed upstream and customized architectures, limiting their adaptability. To address these limitations, we propose Offline-Poly, a general offline 3D MOT method based on a tracking-centric design. We introduce a standardized paradigm termed Tracking-by-Tracking (TBT), which operates exclusively on arbitrary off-the-shelf tracking outputs and produces offline-refined tracklets. This formulation decouples offline tracker from specific upstream detectors or trackers. Under the TBT paradigm, Offline-Poly accepts one or multiple coarse tracking results and processes them through a structured pipeline comprising pre-processing, hierarchical matching and fusion, and tracklet refinement. Each module is designed to capitalize on the two fundamental properties of offline tracking: resource unconstrainedness, which permits global optimization beyond real-time limits, and future observability, which enables tracklet reasoning over the full temporal horizon. Offline-Poly first eliminates short-term ghost tracklets and re-identifies fragmented segments using global scene context. It then constructs scene-level similarity to associate tracklets across multiple input sources. Finally, Offline-Poly refines tracklets by jointly leveraging local and global motion patterns. On nuScenes, we achieve SOTA performance with 77.6% AMOTA. On KITTI, it achieves leading results with 83.00% HOTA. Comprehensive experiments further validate the flexibility, generalizability, and modular effectiveness of Offline-Poly.