CGCV:Context Guided Correlation Volume for Optical Flow Neural Networks

TL;DR

This paper introduces CGCV, a novel context-guided correlation volume for optical flow neural networks, improving robustness against noise, motion blur, and atmospheric effects, and enhancing existing models like RAFT and GMA.

Contribution

We propose a new CGCV method that integrates context information into correlation volume computation, significantly boosting optical flow accuracy and robustness with minimal additional parameters.

Findings

CGCV improves optical flow performance in challenging conditions.

Integrating CGCV with GMA enhances leader board rankings.

Our correlation volume outperforms some transformer-based models.

Abstract

Optical flow, which computes the apparent motion from a pair of video frames, is a critical tool for scene motion estimation. Correlation volume is the central component of optical flow computational neural models. It estimates the pairwise matching costs between cross-frame features, and is then used to decode optical flow. However, traditional correlation volume is frequently noisy, outlier-prone, and sensitive to motion blur. We observe that, although the recent RAFT algorithm also adopts the traditional correlation volume, its additional context encoder provides semantically representative features to the flow decoder, implicitly compensating for the deficiency of the correlation volume. However, the benefits of this context encoder has been barely discussed or exploited. In this paper, we first investigate the functionality of RAFT's context encoder, then propose a new Context Guided Correlation Volume (CGCV) via gating and lifting schemes. CGCV can be universally integrated with RAFT-based flow computation methods for enhanced performance, especially effective in the presence of motion blur, de-focus blur and atmospheric effects. By incorporating the proposed CGCV with previous Global Motion Aggregation (GMA) method, at a minor cost of 0.5% extra parameters, the rank of GMA is lifted by 23 places on KITTI 2015 Leader Board, and 3 places on Sintel Leader Board. Moreover, at a similar model size, our correlation volume achieves competitive or superior performance to state of the art peer supervised models that employ Transformers or Graph Reasoning, as verified by extensive experiments.