Learning Optical Flow from a Few Matches

TL;DR

This paper introduces a sparse correlation volume for optical flow estimation, significantly reducing computation and memory requirements while maintaining high accuracy, by using only the most relevant feature matches instead of dense correlation volumes.

Contribution

It proposes a novel sparse correlation volume representation that replaces dense volumes, enabling efficient optical flow estimation with less computation and memory.

Findings

Reduces computational cost and memory usage significantly.

Maintains high accuracy comparable to dense correlation volume methods.

Demonstrates effectiveness on optical flow benchmarks.

Abstract

State-of-the-art neural network models for optical flow estimation require a dense correlation volume at high resolutions for representing per-pixel displacement. Although the dense correlation volume is informative for accurate estimation, its heavy computation and memory usage hinders the efficient training and deployment of the models. In this paper, we show that the dense correlation volume representation is redundant and accurate flow estimation can be achieved with only a fraction of elements in it. Based on this observation, we propose an alternative displacement representation, named Sparse Correlation Volume, which is constructed directly by computing the k closest matches in one feature map for each feature vector in the other feature map and stored in a sparse data structure. Experiments show that our method can reduce computational cost and memory use significantly, while maintaining high accuracy compared to previous approaches with dense correlation volumes. Code is available at https://github.com/zacjiang/scv .