Tracking and Segmenting Anything in Any Modality

TL;DR

This paper introduces SATA, a universal framework for tracking and segmentation across any modality, addressing cross-modal and cross-task challenges to improve generalization in video understanding.

Contribution

SATA unifies multiple tracking and segmentation tasks with any modality input using a Decoupled Mixture-of-Expert mechanism and a Task-aware Multi-object Tracking pipeline, enhancing flexibility and generalization.

Findings

Outperforms on 18 benchmarks in tracking and segmentation.

Effectively handles multiple modalities and tasks.

Improves cross-task and cross-modal knowledge sharing.

Abstract

Tracking and segmentation play essential roles in video understanding, providing basic positional information and temporal association of objects within video sequences. Despite their shared objective, existing approaches often tackle these tasks using specialized architectures or modality-specific parameters, limiting their generalization and scalability. Recent efforts have attempted to unify multiple tracking and segmentation subtasks from the perspectives of any modality input or multi-task inference. However, these approaches tend to overlook two critical challenges: the distributional gap across different modalities and the feature representation gap across tasks. These issues hinder effective cross-task and cross-modal knowledge sharing, ultimately constraining the development of a true generalist model. To address these limitations, we propose a universal tracking and segmentation framework named SATA, which unifies a broad spectrum of tracking and segmentation subtasks with any modality input. Specifically, a Decoupled Mixture-of-Expert (DeMoE) mechanism is presented to decouple the unified representation learning task into the modeling process of cross-modal shared knowledge and specific information, thus enabling the model to maintain flexibility while enhancing generalization. Additionally, we introduce a Task-aware Multi-object Tracking (TaMOT) pipeline to unify all the task outputs as a unified set of instances with calibrated ID information, thereby alleviating the degradation of task-specific knowledge during multi-task training. SATA demonstrates superior performance on 18 challenging tracking and segmentation benchmarks, offering a novel perspective for more generalizable video understanding.