Online Reasoning Video Object Segmentation

TL;DR

This paper introduces ORVOS, a new benchmark and baseline for online, causal video object segmentation from natural language queries, emphasizing real-time reasoning and referent shifts.

Contribution

It presents the ORVOSB benchmark with causal annotations and proposes a baseline model with continual prompt updates and temporal reasoning capabilities.

Findings

Existing methods perform poorly under causal constraints.

The proposed baseline improves long-horizon reasoning.

ORVOSB enables evaluation of online reasoning in video segmentation.

Abstract

Reasoning video object segmentation predicts pixel-level masks in videos from natural-language queries that may involve implicit and temporally grounded references. However, existing methods are developed and evaluated in an offline regime, where the entire video is available at inference time and future frames can be exploited for retrospective disambiguation, deviating from real-world deployments that require strictly causal, frame-by-frame decisions. We study Online Reasoning Video Object Segmentation (ORVOS), where models must incrementally interpret queries using only past and current frames without revisiting previous predictions, while handling referent shifts as events unfold. To support evaluation, we introduce ORVOSB, a benchmark with frame-level causal annotations and referent-shift labels, comprising 210 videos, 12,907 annotated frames, and 512 queries across five reasoning categories. We further propose a baseline with continually-updated segmentation prompts and a structured temporal token reservoir for long-horizon reasoning under bounded computation. Experiments show that existing methods struggle under strict causality and referent shifts, while our baseline establishes a strong foundation for future research.