Holistic Prototype Attention Network for Few-Shot VOS

TL;DR

This paper introduces HPAN, a novel network for few-shot video object segmentation that leverages prototype graph and bidirectional attention modules to improve segmentation accuracy by capturing inter-frame and inter-class correlations.

Contribution

The paper proposes a holistic prototype attention network with PGAM and BPAM modules, enhancing feature representation and inter-frame consistency for FSVOS.

Findings

HPAN outperforms existing methods on YouTube-FSVOS dataset.

The prototype graph attention module improves local prototype representations.

Bidirectional attention module enhances support-query and intra-frame consistency.

Abstract

Few-shot video object segmentation (FSVOS) aims to segment dynamic objects of unseen classes by resorting to a small set of support images that contain pixel-level object annotations. Existing methods have demonstrated that the domain agent-based attention mechanism is effective in FSVOS by learning the correlation between support images and query frames. However, the agent frame contains redundant pixel information and background noise, resulting in inferior segmentation performance. Moreover, existing methods tend to ignore inter-frame correlations in query videos. To alleviate the above dilemma, we propose a holistic prototype attention network (HPAN) for advancing FSVOS. Specifically, HPAN introduces a prototype graph attention module (PGAM) and a bidirectional prototype attention module (BPAM), transferring informative knowledge from seen to unseen classes. PGAM generates local prototypes from all foreground features and then utilizes their internal correlations to enhance the representation of the holistic prototypes. BPAM exploits the holistic information from support images and video frames by fusing co-attention and self-attention to achieve support-query semantic consistency and inner-frame temporal consistency. Extensive experiments on YouTube-FSVOS have been provided to demonstrate the effectiveness and superiority of our proposed HPAN method.