LLMTrack: Semantic Multi-Object Tracking with Multi-modal Large Language Models

TL;DR

This paper introduces LLMTrack, a novel framework integrating multi-modal large language models into semantic multi-object tracking, enhancing dynamic semantic reasoning and bridging perceptual tracking with cognitive understanding.

Contribution

We propose LLMTrack with a Macro-Understanding-First paradigm and a Spatio-Temporal Fusion Module, pioneering the integration of MLLMs into SMOT and establishing a new foundation for video understanding.

Findings

Achieves state-of-the-art geometric tracking performance

Significantly improves semantic reasoning capabilities

High-quality semantic narratives enable natural deduction of social interactions

Abstract

Multi-Object Tracking (MOT) is evolving from geometric localization to Semantic MOT (SMOT) to answer complex relational queries, yet progress is hindered by semantic data scarcity and a structural disconnect between tracking architectures and Multi-modal Large Language Models (MLLMs). To address this, we introduce Grand-SMOT, a large-scale, open-world benchmark providing high-density, dual-stream narratives that comprehensively decouple individual behaviors from environmental contexts. Furthermore, we propose LLMTrack, the first framework to seamlessly integrate MLLMs into the SMOT task. LLMTrack establishes a Macro-Understanding-First paradigm, utilizing a novel Spatio-Temporal Fusion Module to align discrete geometric trajectories with continuous semantic features, effectively suppressing temporal hallucinations during online processing. Extensive experiments demonstrate that LLMTrack achieves state-of-the-art geometric tracking performance while delivering a qualitative leap in dynamic semantic reasoning. Notably, our analysis reveals that high-quality semantic narratives empower the language model to deduce complex social interactions naturally, demonstrating that direct cognitive reasoning is more effective than cumbersome explicit visual modeling. Ultimately, our contributions bridge the gap between perceptual tracking and cognitive reasoning, establishing a robust new foundation for comprehensive video understanding and intelligent narrative generation.