Unsupervised Video Object Segmentation via Prototype Memory Network

TL;DR

This paper introduces a prototype memory network for unsupervised video object segmentation, leveraging superpixel-based prototypes and a self-learning mechanism to improve long-range temporal consistency and achieve state-of-the-art results.

Contribution

The paper presents a novel prototype memory network architecture that adaptively stores useful prototypes for better long-range object association in unsupervised video segmentation.

Findings

Achieves state-of-the-art performance on three datasets.

Effective extraction of RGB and motion features via superpixel prototypes.

Self-learning algorithm improves prototype usefulness and memory management.

Abstract

Unsupervised video object segmentation aims to segment a target object in the video without a ground truth mask in the initial frame. This challenging task requires extracting features for the most salient common objects within a video sequence. This difficulty can be solved by using motion information such as optical flow, but using only the information between adjacent frames results in poor connectivity between distant frames and poor performance. To solve this problem, we propose a novel prototype memory network architecture. The proposed model effectively extracts the RGB and motion information by extracting superpixel-based component prototypes from the input RGB images and optical flow maps. In addition, the model scores the usefulness of the component prototypes in each frame based on a self-learning algorithm and adaptively stores the most useful prototypes in memory and discards obsolete prototypes. We use the prototypes in the memory bank to predict the next query frames mask, which enhances the association between distant frames to help with accurate mask prediction. Our method is evaluated on three datasets, achieving state-of-the-art performance. We prove the effectiveness of the proposed model with various ablation studies.