ORMOT: A Dataset and Framework for Omnidirectional Referring Multi-Object Tracking

TL;DR

This paper introduces ORMOT, a new task and dataset for tracking objects in omnidirectional videos based on language descriptions, addressing the limitations of conventional camera-based datasets.

Contribution

The work presents a novel omnidirectional RMOT task, constructs the ORSet dataset with diverse scenes and annotations, and proposes the ORTrack framework leveraging large vision-language models.

Findings

ORTrack outperforms baseline methods on ORSet.

The dataset contains 27 scenes, 848 descriptions, and 3,401 objects.

Omnidirectional imagery improves long-horizon language understanding.

Abstract

Multi-Object Tracking (MOT) is a fundamental task in computer vision, aiming to track targets across video frames. Existing MOT methods perform well in general visual scenes, but face significant challenges and limitations when extended to visual-language settings. To bridge this gap, the task of Referring Multi-Object Tracking (RMOT) has recently been proposed, which aims to track objects that correspond to language descriptions. However, current RMOT methods are primarily developed on datasets captured by conventional cameras, which suffer from limited field of view. This constraint often causes targets to move out of the frame, leading to fragmented tracking and loss of contextual information. In this work, we propose a novel task, called Omnidirectional Referring Multi-Object Tracking (ORMOT), which extends RMOT to omnidirectional imagery, aiming to overcome the field-of-view (FoV) limitation of conventional datasets and improve the model's ability to understand long-horizon language descriptions. To advance the ORMOT task, we construct ORSet, an Omnidirectional Referring Multi-Object Tracking dataset, which contains 27 diverse omnidirectional scenes, 848 language descriptions, and 3,401 annotated objects, providing rich visual, temporal, and language information. Furthermore, we propose ORTrack, a Large Vision-Language Model (LVLM)-driven framework tailored for Omnidirectional Referring Multi-Object Tracking. Extensive experiments on the ORSet dataset demonstrate the effectiveness of our ORTrack framework. The dataset and code will be open-sourced at https://github.com/chen-si-jia/ORMOT.