AgentRVOS: Reasoning over Object Tracks for Zero-Shot Referring Video Object Segmentation

TL;DR

AgentRVOS introduces a training-free, agentic pipeline that combines SAM3 and MLLM for improved zero-shot referring video object segmentation by leveraging object-level evidence and temporal reasoning.

Contribution

It proposes a novel training-free approach that integrates SAM3 and MLLM for better spatio-temporal reasoning in RVOS tasks.

Findings

Achieves state-of-the-art performance among training-free methods

Demonstrates consistent results across various MLLM backbones

Effectively leverages object-level evidence for improved segmentation

Abstract

Referring Video Object Segmentation (RVOS) aims to segment a target object throughout a video given a natural language query. Training-free methods for this task follow a common pipeline: a MLLM selects keyframes, grounds the referred object within those frames, and a video segmentation model propagates the results. While intuitive, this design asks the MLLM to make temporal decisions before any object-level evidence is available, limiting both reasoning quality and spatio-temporal coverage. To overcome this, we propose AgentRVOS, a training-free agentic pipeline built on the complementary strengths of SAM3 and a MLLM. Given a concept derived from the query, SAM3 provides reliable perception over the full spatio-temporal extent through generated mask tracks. The MLLM then identifies the target through query-grounded reasoning over this object-level evidence, iteratively pruning guided by SAM3's temporal existence information. Extensive experiments show that AgentRVOS achieves state-of-the-art performance among training-free methods across multiple benchmarks, with consistent results across diverse MLLM backbones. Our project page is available at: https://cvlab-kaist.github.io/AgentRVOS/.