RCoT-Seg: Reinforced Chain-of-Thought for Video Reasoning and Segmentation

TL;DR

RCoT-Seg introduces a novel video reasoning and segmentation framework that explicitly separates temporal reasoning from spatial perception, improving accuracy and robustness in complex multi-object video scenes.

Contribution

It proposes a reinforced chain-of-thought approach with a keyframe selection module and a two-stage segmentation process, enhancing holistic temporal understanding and spatial precision.

Findings

Achieves state-of-the-art performance on video reasoning and segmentation benchmarks.

Improves moment localization and inter-frame mask consistency.

Effectively separates temporal reasoning from spatial perception for better results.

Abstract

Video Reasoning Segmentation (VRS) aims to segment target objects in videos based on implicit instructions that convey human intent and temporal logic. Existing MLLM-based methods predict masks with a [SEG] token after selecting frames via simple sampling or an auxiliary MLLM, where limited supervision and frame-language similarity rules often yield narrow-scope keyframe choices that weaken holistic temporal understanding and lead to brittle localization in complex multi-object scenes. To address these issues, we introduce RCoT-Seg, a video-of-thought framework that factorizes VRS into temporal video reasoning (TVR) and keyframe target perception (KTP), explicitly separating temporal reasoning from spatial perception. Specifically, in the TVR stage, an agentic keyframe selection module, initialized with a curated CoT-start corpus and refined by GRPO under task-aligned rewards, is proposed to generate and reselect the keyframe through self-evaluation, strengthening moment localization and temporal reasoning. In the KTP stage, RCoT-Seg performs high-resolution segmentation on the selected frame and propagates masks with SAM2-based methods across the sequence, replacing heuristic sampling and external selectors while improving spatial precision and inter-frame consistency. Extensive experimental results demonstrate that the proposed RCoT-Seg achieves favorable performance against the state-of-the-art methods. The code and models will be publicly released at https://github.com/Victor-wjw/RCoT-Seg.