SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warping

TL;DR

SMURF is an unsupervised optical flow method that significantly outperforms previous approaches and even some supervised models by integrating RAFT architecture with novel self-supervision and multi-frame learning techniques.

Contribution

It introduces a new unsupervised learning framework for optical flow that combines RAFT architecture with sequence-aware self-supervision and multi-frame data utilization.

Findings

Improves state-of-the-art by 36-40% on benchmarks.

Outperforms several supervised methods like PWC-Net and FlowNet2.

Effective learning from multi-frame videos with only two frames at inference.

Abstract

We present SMURF, a method for unsupervised learning of optical flow that improves state of the art on all benchmarks by $36\%$ to $40\%$ (over the prior best method UFlow) and even outperforms several supervised approaches such as PWC-Net and FlowNet2. Our method integrates architecture improvements from supervised optical flow, i.e. the RAFT model, with new ideas for unsupervised learning that include a sequence-aware self-supervision loss, a technique for handling out-of-frame motion, and an approach for learning effectively from multi-frame video data while still only requiring two frames for inference.