InterRVOS: Interaction-aware Referring Video Object Segmentation

TL;DR

This paper introduces InterRVOS, a new task and dataset for interaction-aware video object segmentation, emphasizing the separation of actor and target objects in videos described by natural language, and proposes a novel architecture for this purpose.

Contribution

The paper proposes a new interaction-aware RVOS task, creates a large-scale dataset with interaction annotations, and develops a specialized model that improves role-specific segmentation performance.

Findings

ReVIOSa outperforms existing baselines on the new dataset.

The dataset contains over 127K annotated expressions with distinct actor and target masks.

The model achieves strong results on standard RVOS benchmarks.

Abstract

Referring video object segmentation (RVOS) aims to segment objects in a video described by a natural language expression. However, most existing approaches focus on segmenting only the referred object (typically the actor), even when the expression clearly describes an interaction involving multiple objects with distinct roles. For instance, "A throwing B" implies a directional interaction, but standard RVOS segments only the actor (A), neglecting other involved target objects (B). In this paper, we introduce Interaction-aware Referring Video Object Segmentation (InterRVOS), a novel task that focuses on the modeling of interactions. It requires the model to segment the actor and target objects separately, reflecting their asymmetric roles in an interaction. This task formulation enables fine-grained understanding of object relationships, as many video events are defined by such relationships rather than individual objects. To support this task, we propose a new evaluation protocol that separately evaluates actor and target segmentation, enabling more accurate assessment of the model's ability to distinguish and segment actor and target roles. We also present InterRVOS-127K, a large-scale dataset with over 127K automatically annotated expressions, including interaction expressions annotated with distinct masks for actor and target objects. Furthermore, we develop ReVIOSa, an MLLM-based architecture that introduces interaction-aware special tokens and leverages an attention mask loss to enhance role-specific segmentation. Extensive experiments show that ReVIOSa not only outperforms existing baselines on our proposed InterRVOS-127K evaluation set, but also achieves strong performance on standard RVOS benchmarks. Our project page is available at: https://cvlab-kaist.github.io/InterRVOS.