StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences

TL;DR

StreamFlow introduces a fast, efficient multi-frame optical flow estimation method that effectively handles occlusions in video sequences by eliminating redundant computations and modeling spatio-temporal relations during encoding and decoding.

Contribution

The paper proposes a novel in-batch multi-frame optical flow framework with integrated spatio-temporal modeling modules that improve efficiency and accuracy over prior recursive methods.

Findings

Achieves similar speed to two-frame networks while utilizing multi-frame information.

Significantly improves optical flow accuracy in occluded regions.

Reduces computational complexity by 63.82% compared to previous multi-frame methods.

Abstract

Occlusions between consecutive frames have long posed a significant challenge in optical flow estimation. The inherent ambiguity introduced by occlusions directly violates the brightness constancy constraint and considerably hinders pixel-to-pixel matching. To address this issue, multi-frame optical flow methods leverage adjacent frames to mitigate the local ambiguity. Nevertheless, prior multi-frame methods predominantly adopt recursive flow estimation, resulting in a considerable computational overlap. In contrast, we propose a streamlined in-batch framework that eliminates the need for extensive redundant recursive computations while concurrently developing effective spatio-temporal modeling approaches under in-batch estimation constraints. Specifically, we present a Streamlined In-batch Multi-frame (SIM) pipeline tailored to video input, attaining a similar level of time efficiency to two-frame networks. Furthermore, we introduce an efficient Integrative Spatio-temporal Coherence (ISC) modeling method for effective spatio-temporal modeling during the encoding phase, which introduces no additional parameter overhead. Additionally, we devise a Global Temporal Regressor (GTR) that effectively explores temporal relations during decoding. Benefiting from the efficient SIM pipeline and effective modules, StreamFlow not only excels in terms of performance on the challenging KITTI and Sintel datasets, with particular improvement in occluded areas but also attains a remarkable $63.82\%$ enhancement in speed compared with previous multi-frame methods. The code will be available soon at https://github.com/littlespray/StreamFlow.