Memory-Efficient Optical Flow via Radius-Distribution Orthogonal Cost Volume

TL;DR

MeFlow introduces a memory-efficient optical flow estimation method for high-resolution images by decomposing the search space into orthogonal components and using self-attention, enabling scalability and competitive accuracy.

Contribution

The paper proposes a novel recurrent local orthogonal cost volume representation and radius-distribution multi-scale lookup for efficient high-resolution optical flow estimation.

Findings

Achieves high accuracy on Sintel and KITTI benchmarks.

Maintains the highest memory efficiency on high-resolution inputs.

Effective for real-world 4K images.

Abstract

The full 4D cost volume in Recurrent All-Pairs Field Transforms (RAFT) or global matching by Transformer achieves impressive performance for optical flow estimation. However, their memory consumption increases quadratically with input resolution, rendering them impractical for high-resolution images. In this paper, we present MeFlow, a novel memory-efficient method for high-resolution optical flow estimation. The key of MeFlow is a recurrent local orthogonal cost volume representation, which decomposes the 2D search space dynamically into two 1D orthogonal spaces, enabling our method to scale effectively to very high-resolution inputs. To preserve essential information in the orthogonal space, we utilize self attention to propagate feature information from the 2D space to the orthogonal space. We further propose a radius-distribution multi-scale lookup strategy to model the correspondences of large displacements at a negligible cost. We verify the efficiency and effectiveness of our method on the challenging Sintel and KITTI benchmarks, and real-world 4K ($2160\!\times\!3840$) images. Our method achieves competitive performance on both Sintel and KITTI benchmarks, while maintaining the highest memory efficiency on high-resolution inputs.