SplatFlow: Learning Multi-frame Optical Flow via Splatting

TL;DR

SplatFlow introduces a multi-frame optical flow estimation framework that effectively handles occlusions using differentiable splatting, outperforming existing methods on KITTI2015 and Sintel benchmarks.

Contribution

It proposes a novel differentiable splatting transformation and a Final-to-All embedding method to improve multi-frame optical flow estimation, especially in occluded regions.

Findings

Outperforms all published methods on KITTI2015 and Sintel benchmarks.

Achieves 19.4% and 16.2% error reductions on Sintel for clean and final passes.

Handles occlusions more effectively than previous approaches.

Abstract

The occlusion problem remains a crucial challenge in optical flow estimation (OFE). Despite the recent significant progress brought about by deep learning, most existing deep learning OFE methods still struggle to handle occlusions; in particular, those based on two frames cannot correctly handle occlusions because occluded regions have no visual correspondences. However, there is still hope in multi-frame settings, which can potentially mitigate the occlusion issue in OFE. Unfortunately, multi-frame OFE (MOFE) remains underexplored, and the limited studies on it are mainly specially designed for pyramid backbones or else obtain the aligned previous frame's features, such as correlation volume and optical flow, through time-consuming backward flow calculation or non-differentiable forward warping transformation. This study proposes an efficient MOFE framework named SplatFlow to address these shortcomings. SplatFlow introduces the differentiable splatting transformation to align the previous frame's motion feature and designs a Final-to-All embedding method to input the aligned motion feature into the current frame's estimation, thus remodeling the existing two-frame backbones. The proposed SplatFlow is efficient yet more accurate, as it can handle occlusions properly. Extensive experimental evaluations show that SplatFlow substantially outperforms all published methods on the KITTI2015 and Sintel benchmarks. Especially on the Sintel benchmark, SplatFlow achieves errors of 1.12 (clean pass) and 2.07 (final pass), with surprisingly significant 19.4% and 16.2% error reductions, respectively, from the previous best results submitted. The code for SplatFlow is available at https://github.com/wwsource/SplatFlow.