SLAck: Semantic, Location, and Appearance Aware Open-Vocabulary Tracking

TL;DR

SLAck introduces a unified framework that integrates semantics, location, and appearance cues early in the association process for open-vocabulary multiple object tracking, significantly improving performance on large-scale benchmarks.

Contribution

It proposes a novel joint consideration of multiple cues via a lightweight graph, eliminating heuristic post-processing and enhancing open-vocabulary tracking accuracy.

Findings

Outperforms previous state-of-the-art on open-vocabulary MOT and TAO TETA benchmarks.

Effectively integrates semantics, location, and appearance cues early in tracking.

Reduces reliance on complex heuristics, simplifying the tracking pipeline.

Abstract

Open-vocabulary Multiple Object Tracking (MOT) aims to generalize trackers to novel categories not in the training set. Currently, the best-performing methods are mainly based on pure appearance matching. Due to the complexity of motion patterns in the large-vocabulary scenarios and unstable classification of the novel objects, the motion and semantics cues are either ignored or applied based on heuristics in the final matching steps by existing methods. In this paper, we present a unified framework SLAck that jointly considers semantics, location, and appearance priors in the early steps of association and learns how to integrate all valuable information through a lightweight spatial and temporal object graph. Our method eliminates complex post-processing heuristics for fusing different cues and boosts the association performance significantly for large-scale open-vocabulary tracking. Without bells and whistles, we outperform previous state-of-the-art methods for novel classes tracking on the open-vocabulary MOT and TAO TETA benchmarks. Our code is available at \href{https://github.com/siyuanliii/SLAck}{github.com/siyuanliii/SLAck}.