STORM: End-to-End Referring Multi-Object Tracking in Videos

TL;DR

STORM is an end-to-end multi-object tracking framework that jointly performs grounding and tracking, leveraging a new dataset and a task-composition learning strategy to improve performance and data efficiency.

Contribution

The paper introduces STORM, a unified model for referring multi-object tracking, and STORM-Bench, a new dataset with accurate trajectories and diverse referring expressions.

Findings

Achieves state-of-the-art results on multiple benchmarks.

Demonstrates strong generalization and robust spatial-temporal reasoning.

Leverages a task-composition learning strategy for improved data efficiency.

Abstract

Referring multi-object tracking (RMOT) is a task of associating all the objects in a video that semantically match with given textual queries or referring expressions. Existing RMOT approaches decompose object grounding and tracking into separated modules and exhibit limited performance due to the scarcity of training videos, ambiguous annotations, and restricted domains. In this work, we introduce STORM, an end-to-end MLLM that jointly performs grounding and tracking within a unified framework, eliminating external detectors and enabling coherent reasoning over appearance, motion, and language. To improve data efficiency, we propose a task-composition learning (TCL) strategy that decomposes RMOT into image grounding and object tracking, allowing STORM to leverage data-rich sub-tasks and learn structured spatial--temporal reasoning. We further construct STORM-Bench, a new RMOT dataset with accurate trajectories and diverse, unambiguous referring expressions generated through a bottom-up annotation pipeline. Extensive experiments show that STORM achieves state-of-the-art performance on image grounding, single-object tracking, and RMOT benchmarks, demonstrating strong generalization and robust spatial--temporal grounding in complex real-world scenarios. STORM-Bench is released at https://github.com/amazon-science/storm-referring-multi-object-grounding.