UniRef++: Segment Every Reference Object in Spatial and Temporal Spaces

TL;DR

UniRef++ introduces a unified architecture for multiple reference-based object segmentation tasks, enabling multi-task learning and achieving state-of-the-art results across various benchmarks.

Contribution

The paper proposes UniRef++, a single architecture with UniFusion for unified multi-task object segmentation, allowing flexible task execution and efficient training.

Findings

Achieves state-of-the-art on RIS and RVOS benchmarks.

Performs competitively on FSS and VOS with shared parameters.

Incorporates UniFusion into foundation models like SAM for efficient finetuning.

Abstract

The reference-based object segmentation tasks, namely referring image segmentation (RIS), few-shot image segmentation (FSS), referring video object segmentation (RVOS), and video object segmentation (VOS), aim to segment a specific object by utilizing either language or annotated masks as references. Despite significant progress in each respective field, current methods are task-specifically designed and developed in different directions, which hinders the activation of multi-task capabilities for these tasks. In this work, we end the current fragmented situation and propose UniRef++ to unify the four reference-based object segmentation tasks with a single architecture. At the heart of our approach is the proposed UniFusion module which performs multiway-fusion for handling different tasks with respect to their specified references. And a unified Transformer architecture is then adopted for achieving instance-level segmentation. With the unified designs, UniRef++ can be jointly trained on a broad range of benchmarks and can flexibly complete multiple tasks at run-time by specifying the corresponding references. We evaluate our unified models on various benchmarks. Extensive experimental results indicate that our proposed UniRef++ achieves state-of-the-art performance on RIS and RVOS, and performs competitively on FSS and VOS with a parameter-shared network. Moreover, we showcase that the proposed UniFusion module could be easily incorporated into the current advanced foundation model SAM and obtain satisfactory results with parameter-efficient finetuning. Codes and models are available at \url{https://github.com/FoundationVision/UniRef}.