RMem: Restricted Memory Banks Improve Video Object Segmentation

TL;DR

This paper introduces RMem, a simple approach that restricts memory bank size in video object segmentation, leading to improved accuracy and state-of-the-art results by reducing confusion from redundant information.

Contribution

The paper reveals that limiting memory banks to essential frames enhances VOS performance and introduces the concept of temporal positional embedding, improving temporal reasoning.

Findings

Restricted memory banks improve VOS accuracy.

RMem achieves state-of-the-art results on VOST and Long Videos datasets.

Restricting memory reduces training-inference discrepancy.

Abstract

With recent video object segmentation (VOS) benchmarks evolving to challenging scenarios, we revisit a simple but overlooked strategy: restricting the size of memory banks. This diverges from the prevalent practice of expanding memory banks to accommodate extensive historical information. Our specially designed "memory deciphering" study offers a pivotal insight underpinning such a strategy: expanding memory banks, while seemingly beneficial, actually increases the difficulty for VOS modules to decode relevant features due to the confusion from redundant information. By restricting memory banks to a limited number of essential frames, we achieve a notable improvement in VOS accuracy. This process balances the importance and freshness of frames to maintain an informative memory bank within a bounded capacity. Additionally, restricted memory banks reduce the training-inference discrepancy in memory lengths compared with continuous expansion. This fosters new opportunities in temporal reasoning and enables us to introduce the previously overlooked "temporal positional embedding." Finally, our insights are embodied in "RMem" ("R" for restricted), a simple yet effective VOS modification that excels at challenging VOS scenarios and establishes new state of the art for object state changes (on the VOST dataset) and long videos (on the Long Videos dataset). Our code and demo are available at https://restricted-memory.github.io/.