DeVOS: Flow-Guided Deformable Transformer for Video Object Segmentation

TL;DR

DeVOS introduces a novel deformable transformer architecture for video object segmentation that leverages motion-guided attention and optical flow to enhance temporal consistency and robustness in long-term tracking.

Contribution

The paper proposes DeVOS, combining memory-based matching with motion-guided deformable attention, improving long-term video object segmentation performance and stability.

Findings

Achieves top performance on DAVIS 2017 and YouTube-VOS datasets.

Introduces ADVA for adaptive local similarity search.

Maintains consistent runtime and memory efficiency.

Abstract

The recent works on Video Object Segmentation achieved remarkable results by matching dense semantic and instance-level features between the current and previous frames for long-time propagation. Nevertheless, global feature matching ignores scene motion context, failing to satisfy temporal consistency. Even though some methods introduce local matching branch to achieve smooth propagation, they fail to model complex appearance changes due to the constraints of the local window. In this paper, we present DeVOS (Deformable VOS), an architecture for Video Object Segmentation that combines memory-based matching with motion-guided propagation resulting in stable long-term modeling and strong temporal consistency. For short-term local propagation, we propose a novel attention mechanism ADVA (Adaptive Deformable Video Attention), allowing the adaption of similarity search region to query-specific semantic features, which ensures robust tracking of complex shape and scale changes. DeVOS employs an optical flow to obtain scene motion features which are further injected to deformable attention as strong priors to learnable offsets. Our method achieves top-rank performance on DAVIS 2017 val and test-dev (88.1%, 83.0%), YouTube-VOS 2019 val (86.6%) while featuring consistent run-time speed and stable memory consumption