Integrating Boxes and Masks: A Multi-Object Framework for Unified Visual Tracking and Segmentation

TL;DR

This paper introduces MITS, a unified framework that effectively combines object tracking and segmentation for multiple objects, supporting both box and mask initialization, and achieves state-of-the-art results on VOT and VOS benchmarks.

Contribution

The paper presents a novel multi-object framework that integrates boxes and masks for unified tracking and segmentation, with a new identification module and pinpoint box predictor.

Findings

MITS surpasses previous VOT methods by around 6% on GOT-10k.

It significantly improves box initialization performance on VOS benchmarks.

Achieves state-of-the-art results on both VOT and VOS datasets.

Abstract

Tracking any given object(s) spatially and temporally is a common purpose in Visual Object Tracking (VOT) and Video Object Segmentation (VOS). Joint tracking and segmentation have been attempted in some studies but they often lack full compatibility of both box and mask in initialization and prediction, and mainly focus on single-object scenarios. To address these limitations, this paper proposes a Multi-object Mask-box Integrated framework for unified Tracking and Segmentation, dubbed MITS. Firstly, the unified identification module is proposed to support both box and mask reference for initialization, where detailed object information is inferred from boxes or directly retained from masks. Additionally, a novel pinpoint box predictor is proposed for accurate multi-object box prediction, facilitating target-oriented representation learning. All target objects are processed simultaneously from encoding to propagation and decoding, as a unified pipeline for VOT and VOS. Experimental results show MITS achieves state-of-the-art performance on both VOT and VOS benchmarks. Notably, MITS surpasses the best prior VOT competitor by around 6% on the GOT-10k test set, and significantly improves the performance of box initialization on VOS benchmarks. The code is available at https://github.com/yoxu515/MITS.