OmniPT: Unleashing the Potential of Large Vision Language Models for Pedestrian Tracking and Understanding

TL;DR

This paper introduces OmniPT, a unified framework leveraging large vision-language models for pedestrian tracking that combines tracking, referencing, and semantic understanding, showing improved performance over prior methods.

Contribution

The paper presents a novel Pedestrian Tracking framework, OmniPT, integrating tracking, referencing, and semantic understanding using LVLMs with a specialized training pipeline.

Findings

Outperforms previous pedestrian tracking methods on benchmark datasets.

Enables LVLMs to output formatted bounding box and tracking information.

Demonstrates improved semantic understanding of tracked objects.

Abstract

LVLMs have been shown to perform excellently in image-level tasks such as VQA and caption. However, in many instance-level tasks, such as visual grounding and object detection, LVLMs still show performance gaps compared to previous expert models. Meanwhile, although pedestrian tracking is a classical task, there have been a number of new topics in combining object tracking and natural language, such as Referring MOT, Cross-view Referring MOT, and Semantic MOT. These tasks emphasize that models should understand the tracked object at an advanced semantic level, which is exactly where LVLMs excel. In this paper, we propose a new unified Pedestrian Tracking framework, namely OmniPT, which can track, track based on reference and generate semantic understanding of tracked objects interactively. We address two issues: how to model the tracking task into a task that foundation models can perform, and how to make the model output formatted answers. To this end, we implement a training phase consisting of RL-Mid Training-SFT-RL. Based on the pre-trained weights of the LVLM, we first perform a simple RL phase to enable the model to output fixed and supervisable bounding box format. Subsequently, we conduct a mid-training phase using a large number of pedestrian-related datasets. Finally, we perform supervised fine-tuning on several pedestrian tracking datasets, and then carry out another RL phase to improve the model's tracking performance and enhance its ability to follow instructions. We conduct experiments on tracking benchmarks and the experimental results demonstrate that the proposed method can perform better than the previous methods.