Hierarchical Graph Pattern Understanding for Zero-Shot VOS

TL;DR

This paper introduces a hierarchical graph neural network architecture that leverages motion cues and structural relations to improve zero-shot video object segmentation, overcoming limitations of optical flow dependency.

Contribution

The proposed HGPU model innovatively combines hierarchical graph encoding with motion and appearance features for robust zero-shot VOS, achieving state-of-the-art results.

Findings

State-of-the-art performance on four benchmarks

Effective modeling of structural relations improves segmentation

Robustness against optical flow estimation failures

Abstract

The optical flow guidance strategy is ideal for obtaining motion information of objects in the video. It is widely utilized in video segmentation tasks. However, existing optical flow-based methods have a significant dependency on optical flow, which results in poor performance when the optical flow estimation fails for a particular scene. The temporal consistency provided by the optical flow could be effectively supplemented by modeling in a structural form. This paper proposes a new hierarchical graph neural network (GNN) architecture, dubbed hierarchical graph pattern understanding (HGPU), for zero-shot video object segmentation (ZS-VOS). Inspired by the strong ability of GNNs in capturing structural relations, HGPU innovatively leverages motion cues (\ie, optical flow) to enhance the high-order representations from the neighbors of target frames. Specifically, a hierarchical graph pattern encoder with message aggregation is introduced to acquire different levels of motion and appearance features in a sequential manner. Furthermore, a decoder is designed for hierarchically parsing and understanding the transformed multi-modal contexts to achieve more accurate and robust results. HGPU achieves state-of-the-art performance on four publicly available benchmarks (DAVIS-16, YouTube-Objects, Long-Videos and DAVIS-17). Code and pre-trained model can be found at \url{https://github.com/NUST-Machine-Intelligence-Laboratory/HGPU}.