Moving Object Segmentation: All You Need Is SAM (and Flow)

TL;DR

This paper demonstrates that combining the Segment Anything Model (SAM) with optical flow techniques enables simple yet highly effective moving object segmentation in videos, outperforming previous methods across multiple benchmarks.

Contribution

The paper introduces two straightforward methods that integrate SAM with optical flow for motion segmentation, achieving state-of-the-art results without complex training or modifications.

Findings

Outperforms all previous approaches on single-object benchmarks.

Achieves superior results on multi-object moving segmentation tasks.

Maintains object identity in sequence-level segmentation.

Abstract

The objective of this paper is motion segmentation -- discovering and segmenting the moving objects in a video. This is a much studied area with numerous careful, and sometimes complex, approaches and training schemes including: self-supervised learning, learning from synthetic datasets, object-centric representations, amodal representations, and many more. Our interest in this paper is to determine if the Segment Anything model (SAM) can contribute to this task. We investigate two models for combining SAM with optical flow that harness the segmentation power of SAM with the ability of flow to discover and group moving objects. In the first model, we adapt SAM to take optical flow, rather than RGB, as an input. In the second, SAM takes RGB as an input, and flow is used as a segmentation prompt. These surprisingly simple methods, without any further modifications, outperform all previous approaches by a considerable margin in both single and multi-object benchmarks. We also extend these frame-level segmentations to sequence-level segmentations that maintain object identity. Again, this simple model achieves outstanding performance across multiple moving object segmentation benchmarks.