AgentRVOS for MeViS-Text Track of 5th PVUW Challenge: 3rd Method

TL;DR

This paper presents AgentRVOS, a multi-agent pipeline for video object segmentation that integrates semantic hypotheses, agent-driven decision-making, and refinement strategies to improve accuracy and robustness.

Contribution

It introduces a novel multi-agent framework that combines dense semantic hypotheses with agent roles for refined video object segmentation.

Findings

Achieved accurate dense grounded understanding in video segmentation.

Effectively handles presence verification and temporal search.

Improves mask refinement through collaborative agent strategies.

Abstract

This report describes a Ref-VOS pipeline centered on Sa2VA and organized with explicit agent roles. The key idea is that Sa2VA should provide the first dense semantic hypothesis, while an agent loop decides whether that hypothesis should be accepted, revised, or refined. The pipeline starts with a target-presence judgment stage. If the referred object does not exist in the video, the system directly outputs zero masks. Otherwise, Sa2VA receives the video and referring prompt and produces a coarse mask trajectory over the full video. This trajectory is treated as a semantic prior rather than a final answer. A planner agent decomposes the query, temporal partition agents identify informative blocks, scout agents search for anchor frames, and refinement agents convert reliable Sa2VA masks into boxes and points for SAM3 propagation. A critic scores candidate trajectories, a reflection controller repairs weak hypotheses, and a collaboration controller reconciles multiple agent branches. The result is a Ref-VOS system in which Sa2VA is responsible for dense grounded understanding, while the agent layer handles presence verification, temporal search, confidence-aware revision, and final mask refinement.