Training-Free Spatio-temporal Decoupled Reasoning Video Segmentation with Adaptive Object Memory

TL;DR

This paper introduces SDAM, a training-free, spatio-temporal decoupled reasoning framework for video segmentation that uses adaptive object memory to improve stability and accuracy across video sequences without fine-tuning.

Contribution

The paper proposes a novel training-free reasoning framework with an adaptive object memory module and spatio-temporal decoupling for stable, accurate video segmentation.

Findings

Outperforms fine-tuned methods on five benchmark datasets

Achieves precise spatial localization and segmentation

Provides stable cross-frame temporal propagation

Abstract

Reasoning Video Object Segmentation (ReasonVOS) is a challenging task that requires stable object segmentation across video sequences using implicit and complex textual inputs. Previous methods fine-tune Multimodal Large Language Models (MLLMs) to produce segmentation outputs, which demand substantial resources. Additionally, some existing methods are coupled in the processing of spatio-temporal information, which affects the temporal stability of the model to some extent. To address these issues, we propose Training-Free \textbf{S}patio-temporal \textbf{D}ecoupled Reasoning Video Segmentation with \textbf{A}daptive Object \textbf{M}emory (SDAM). We aim to design a training-free reasoning video segmentation framework that outperforms existing methods requiring fine-tuning, using only pre-trained models. Meanwhile, we propose an Adaptive Object Memory module that selects and memorizes key objects based on motion cues in different video sequences. Finally, we propose Spatio-temporal Decoupling for stable temporal propagation. In the spatial domain, we achieve precise localization and segmentation of target objects, while in the temporal domain, we leverage key object temporal information to drive stable cross-frame propagation. Our method achieves excellent results on five benchmark datasets, including Ref-YouTubeVOS, Ref-DAVIS17, MeViS, ReasonVOS, and ReVOS.