Show Me When and Where: Towards Referring Video Object Segmentation in the Wild

TL;DR

This paper introduces a new in-the-wild RVOS setting and dataset, YoURVOS, challenging methods to identify when and where objects appear in untrimmed videos, and proposes OMFormer as a baseline solution.

Contribution

The paper presents a new challenging RVOS benchmark dataset from untrimmed videos and a novel Object-level Multimodal TransFormer (OMFormer) model for improved localization.

Findings

Previous VOS methods perform poorly on YoURVOS, especially with more target-absent frames.

OMFormer consistently outperforms existing methods on the YoURVOS benchmark.

YoURVOS provides a more realistic and challenging testbed for RVOS research.

Abstract

Referring video object segmentation (RVOS) has recently generated great popularity in computer vision due to its widespread applications. Existing RVOS setting contains elaborately trimmed videos, with text-referred objects always appearing in all frames, which however fail to fully reflect the realistic challenges of this task. This simplified setting requires RVOS methods to only predict where objects, with no need to show when the objects appear. In this work, we introduce a new setting towards in-the-wild RVOS. To this end, we collect a new benchmark dataset using Youtube Untrimmed videos for RVOS - YoURVOS, which contains 1,120 in-the-wild videos with 7 times more duration and scenes than existing datasets. Our new benchmark challenges RVOS methods to show not only where but also when objects appear in videos. To set a baseline, we propose Object-level Multimodal TransFormers (OMFormer) to tackle the challenges, which are characterized by encoding object-level multimodal interactions for efficient and global spatial-temporal localisation. We demonstrate that previous VOS methods struggle on our YoURVOS benchmark, especially with the increase of target-absent frames, while our OMFormer consistently performs well. Our YoURVOS dataset offers an imperative benchmark, which will push forward the advancement of RVOS methods for practical applications.