Memory Matching is not Enough: Jointly Improving Memory Matching and Decoding for Video Object Segmentation

TL;DR

This paper introduces a joint approach to improve memory matching and decoding in video object segmentation, effectively reducing false matches and recovering lost information, leading to state-of-the-art results on multiple benchmarks.

Contribution

It proposes a novel cost-aware matching mechanism and a compensatory decoding strategy to enhance memory-based video object segmentation performance.

Findings

Achieves 92.4% on DAVIS 2016 Val

Achieves 88.1% on DAVIS 2017 Val

Achieves 84.8% on YouTubeVOS 2018 Val

Abstract

Memory-based video object segmentation methods model multiple objects over long temporal-spatial spans by establishing memory bank, which achieve the remarkable performance. However, they struggle to overcome the false matching and are prone to lose critical information, resulting in confusion among different objects. In this paper, we propose an effective approach which jointly improving the matching and decoding stages to alleviate the false matching issue.For the memory matching stage, we present a cost aware mechanism that suppresses the slight errors for short-term memory and a shunted cross-scale matching for long-term memory which establish a wide filed matching spaces for various object scales. For the readout decoding stage, we implement a compensatory mechanism aims at recovering the essential information where missing at the matching stage. Our approach achieves the outstanding performance in several popular benchmarks (i.e., DAVIS 2016&2017 Val (92.4%&88.1%), and DAVIS 2017 Test (83.9%)), and achieves 84.8%&84.6% on YouTubeVOS 2018&2019 Val.