Global Motion Understanding in Large-Scale Video Object Segmentation

TL;DR

This paper introduces WarpFormer, a semi-supervised video object segmentation method that leverages global motion knowledge from optical flow to improve accuracy and robustness in complex scenarios, trained on a large-scale dataset.

Contribution

WarpFormer integrates pre-trained optical flow estimation to enhance segmentation propagation, utilizing large-scale training data for improved performance in complex video scenarios.

Findings

Achieves 93.0% on DAVIS 2016 validation

Outperforms state-of-the-art on DAVIS 2017 test-dev

Demonstrates strong results on YouTube-VOS 2019 validation

Abstract

In this paper, we show that transferring knowledge from other domains of video understanding combined with large-scale learning can improve robustness of Video Object Segmentation (VOS) under complex circumstances. Namely, we focus on integrating scene global motion knowledge to improve large-scale semi-supervised Video Object Segmentation. Prior works on VOS mostly rely on direct comparison of semantic and contextual features to perform dense matching between current and past frames, passing over actual motion structure. On the other hand, Optical Flow Estimation task aims to approximate the scene motion field, exposing global motion patterns which are typically undiscoverable during all pairs similarity search. We present WarpFormer, an architecture for semi-supervised Video Object Segmentation that exploits existing knowledge in motion understanding to conduct smoother propagation and more accurate matching. Our framework employs a generic pretrained Optical Flow Estimation network whose prediction is used to warp both past frames and instance segmentation masks to the current frame domain. Consequently, warped segmentation masks are refined and fused together aiming to inpaint occluded regions and eliminate artifacts caused by flow field imperfects. Additionally, we employ novel large-scale MOSE 2023 dataset to train model on various complex scenarios. Our method demonstrates strong performance on DAVIS 2016/2017 validation (93.0% and 85.9%), DAVIS 2017 test-dev (80.6%) and YouTube-VOS 2019 validation (83.8%) that is competitive with alternative state-of-the-art methods while using much simpler memory mechanism and instance understanding logic.