Efficient Correlation Volume Sampling for Ultra-High-Resolution Optical Flow Estimation

TL;DR

This paper introduces a highly efficient implementation of correlation volume sampling for optical flow estimation, significantly reducing memory usage and computation time while maintaining accuracy, enabling high-resolution processing.

Contribution

We propose a novel, efficient correlation volume sampling method that matches the exact operator of RAFT, outperforming existing on-demand sampling in speed and memory efficiency.

Findings

Up to 90% faster than on-demand sampling

Up to 95% lower memory usage compared to default implementation

Achieves state-of-the-art results on 8K ultra-high-resolution datasets

Abstract

Recent optical flow estimation methods often employ local cost sampling from a dense all-pairs correlation volume. This results in quadratic computational and memory complexity in the number of pixels. Although an alternative memory-efficient implementation with on-demand cost computation exists, this is slower in practice and therefore prior methods typically process images at reduced resolutions, missing fine-grained details. To address this, we propose a more efficient implementation of the all-pairs correlation volume sampling, still matching the exact mathematical operator as defined by RAFT. Our approach outperforms on-demand sampling by up to 90% while maintaining low memory usage, and performs on par with the default implementation with up to 95% lower memory usage. As cost sampling makes up a significant portion of the overall runtime, this can translate to up to 50% savings for the total end-to-end model inference in memory-constrained environments. Our evaluation of existing methods includes an 8K ultra-high-resolution dataset and an additional inference-time modification of the recent SEA-RAFT method. With this, we achieve state-of-the-art results at high resolutions both in accuracy and efficiency.