Rethink Predicting the Optical Flow with the Kinetics Perspective

TL;DR

This paper introduces a novel approach to optical flow estimation that combines apparent and kinetic information, improving efficiency and accuracy, especially in occlusion and fast motion scenarios, without relying on correlation volume computation.

Contribution

It proposes a new method that predicts optical flow directly from features, incorporates a differentiable warp considering occlusion, and blends kinetic features with apparent features using a self-supervised loss.

Findings

Outperforms state-of-the-art methods in occlusion scenarios

Achieves better accuracy in fast-moving situations

Reduces computational complexity by avoiding correlation volume

Abstract

Optical flow estimation is one of the fundamental tasks in low-level computer vision, which describes the pixel-wise displacement and can be used in many other tasks. From the apparent aspect, the optical flow can be viewed as the correlation between the pixels in consecutive frames, so continuously refining the correlation volume can achieve an outstanding performance. However, it will make the method have a catastrophic computational complexity. Not only that, the error caused by the occlusion regions of the successive frames will be amplified through the inaccurate warp operation. These challenges can not be solved only from the apparent view, so this paper rethinks the optical flow estimation from the kinetics viewpoint.We propose a method combining the apparent and kinetics information from this motivation. The proposed method directly predicts the optical flow from the feature extracted from images instead of building the correlation volume, which will improve the efficiency of the whole network. Meanwhile, the proposed method involves a new differentiable warp operation that simultaneously considers the warping and occlusion. Moreover, the proposed method blends the kinetics feature with the apparent feature through the novel self-supervised loss function. Furthermore, comprehensive experiments and ablation studies prove that the proposed novel insight into how to predict the optical flow can achieve the better performance of the state-of-the-art methods, and in some metrics, the proposed method outperforms the correlation-based method, especially in situations containing occlusion and fast moving. The code will be public.