SAMFlow: Eliminating Any Fragmentation in Optical Flow with Segment Anything Model

TL;DR

SAMFlow leverages the Segment Anything Model to improve optical flow estimation by enhancing object integrity, achieving state-of-the-art results on Sintel and KITTI-15 benchmarks.

Contribution

The paper introduces a novel integration of SAM with FlowFormer, including task-specific adaptation modules, to address fragmentation in optical flow estimation.

Findings

Achieves top performance on Sintel and KITTI-15 benchmarks.

Surpasses previous methods in EPE and F1-all metrics.

Demonstrates the effectiveness of large vision models in optical flow tasks.

Abstract

Optical Flow Estimation aims to find the 2D dense motion field between two frames. Due to the limitation of model structures and training datasets, existing methods often rely too much on local clues and ignore the integrity of objects, resulting in fragmented motion estimation. Through theoretical analysis, we find the pre-trained large vision models are helpful in optical flow estimation, and we notice that the recently famous Segment Anything Model (SAM) demonstrates a strong ability to segment complete objects, which is suitable for solving the fragmentation problem. We thus propose a solution to embed the frozen SAM image encoder into FlowFormer to enhance object perception. To address the challenge of in-depth utilizing SAM in non-segmentation tasks like optical flow estimation, we propose an Optical Flow Task-Specific Adaption scheme, including a Context Fusion Module to fuse the SAM encoder with the optical flow context encoder, and a Context Adaption Module to adapt the SAM features for optical flow task with Learned Task-Specific Embedding. Our proposed SAMFlow model reaches 0.86/2.10 clean/final EPE and 3.55/12.32 EPE/F1-all on Sintel and KITTI-15 training set, surpassing Flowformer by 8.5%/9.9% and 13.2%/16.3%. Furthermore, our model achieves state-of-the-art performance on the Sintel and KITTI-15 benchmarks, ranking #1 among all two-frame methods on Sintel clean pass.