RAFT: Recurrent All-Pairs Field Transforms for Optical Flow

TL;DR

RAFT introduces a novel recurrent deep network architecture for optical flow that builds multi-scale correlation volumes and iteratively refines flow estimates, achieving state-of-the-art accuracy and efficiency.

Contribution

The paper presents RAFT, a new architecture that significantly improves optical flow estimation by using all-pairs correlation volumes and recurrent updates.

Findings

Achieves 5.10% F1-all error on KITTI, 16% better than previous best.

Obtains 2.855 pixel end-point-error on Sintel, 30% better than previous best.

Demonstrates strong cross-dataset generalization and high efficiency.

Abstract

We introduce Recurrent All-Pairs Field Transforms (RAFT), a new deep network architecture for optical flow. RAFT extracts per-pixel features, builds multi-scale 4D correlation volumes for all pairs of pixels, and iteratively updates a flow field through a recurrent unit that performs lookups on the correlation volumes. RAFT achieves state-of-the-art performance. On KITTI, RAFT achieves an F1-all error of 5.10%, a 16% error reduction from the best published result (6.10%). On Sintel (final pass), RAFT obtains an end-point-error of 2.855 pixels, a 30% error reduction from the best published result (4.098 pixels). In addition, RAFT has strong cross-dataset generalization as well as high efficiency in inference time, training speed, and parameter count. Code is available at https://github.com/princeton-vl/RAFT.