Learning Quality-aware Dynamic Memory for Video Object Segmentation

TL;DR

This paper introduces a Quality-aware Dynamic Memory Network (QDMN) for video object segmentation that selectively stores high-quality frames to prevent error accumulation and dynamically updates memory based on segmentation quality, achieving state-of-the-art results.

Contribution

The paper proposes a novel quality-aware memory mechanism that evaluates and selectively stores frames, improving segmentation accuracy and long-term video processing.

Findings

QDMN achieves state-of-the-art performance on DAVIS and YouTube-VOS benchmarks.

The Quality Assessment Module (QAM) enhances memory-based segmentation methods.

Selective memory updating reduces error accumulation in long videos.

Abstract

Recently, several spatial-temporal memory-based methods have verified that storing intermediate frames and their masks as memory are helpful to segment target objects in videos. However, they mainly focus on better matching between the current frame and the memory frames without explicitly paying attention to the quality of the memory. Therefore, frames with poor segmentation masks are prone to be memorized, which leads to a segmentation mask error accumulation problem and further affect the segmentation performance. In addition, the linear increase of memory frames with the growth of frame number also limits the ability of the models to handle long videos. To this end, we propose a Quality-aware Dynamic Memory Network (QDMN) to evaluate the segmentation quality of each frame, allowing the memory bank to selectively store accurately segmented frames to prevent the error accumulation problem. Then, we combine the segmentation quality with temporal consistency to dynamically update the memory bank to improve the practicability of the models. Without any bells and whistles, our QDMN achieves new state-of-the-art performance on both DAVIS and YouTube-VOS benchmarks. Moreover, extensive experiments demonstrate that the proposed Quality Assessment Module (QAM) can be applied to memory-based methods as generic plugins and significantly improves performance. Our source code is available at https://github.com/workforai/QDMN.