TarViS: A Unified Approach for Target-based Video Segmentation

TL;DR

TarViS introduces a versatile, unified neural network architecture capable of handling multiple video segmentation tasks simultaneously, achieving state-of-the-art results across diverse benchmarks without task-specific retraining.

Contribution

The paper presents TarViS, a novel multi-task video segmentation model that generalizes across various tasks by modeling targets as abstract queries, enabling joint training and task switching during inference.

Findings

Achieves state-of-the-art on 5 out of 7 benchmarks

Successfully applies to four different segmentation tasks

Demonstrates flexible, task-agnostic target modeling

Abstract

The general domain of video segmentation is currently fragmented into different tasks spanning multiple benchmarks. Despite rapid progress in the state-of-the-art, current methods are overwhelmingly task-specific and cannot conceptually generalize to other tasks. Inspired by recent approaches with multi-task capability, we propose TarViS: a novel, unified network architecture that can be applied to any task that requires segmenting a set of arbitrarily defined 'targets' in video. Our approach is flexible with respect to how tasks define these targets, since it models the latter as abstract 'queries' which are then used to predict pixel-precise target masks. A single TarViS model can be trained jointly on a collection of datasets spanning different tasks, and can hot-swap between tasks during inference without any task-specific retraining. To demonstrate its effectiveness, we apply TarViS to four different tasks, namely Video Instance Segmentation (VIS), Video Panoptic Segmentation (VPS), Video Object Segmentation (VOS) and Point Exemplar-guided Tracking (PET). Our unified, jointly trained model achieves state-of-the-art performance on 5/7 benchmarks spanning these four tasks, and competitive performance on the remaining two. Code and model weights are available at: https://github.com/Ali2500/TarViS