Detail Preserving Residual Feature Pyramid Modules for Optical Flow

TL;DR

This paper introduces a Residual Feature Pyramid Module (RFPM) that preserves details in optical flow estimation by reducing blending errors during feature downsampling, leading to improved accuracy and faster training.

Contribution

The novel RFPM module enhances optical flow models by maintaining detail during feature pyramid construction, integrating seamlessly with existing architectures, and enabling efficient transfer learning.

Findings

Reduces flow errors in Sintel and KITTI datasets.

Improves performance of state-of-the-art optical flow methods.

Dramatically decreases training time with transfer learning.

Abstract

Feature pyramids and iterative refinement have recently led to great progress in optical flow estimation. However, downsampling in feature pyramids can cause blending of foreground objects with the background, which will mislead subsequent decisions in the iterative processing. The results are missing details especially in the flow of thin and of small structures. We propose a novel Residual Feature Pyramid Module (RFPM) which retains important details in the feature map without changing the overall iterative refinement design of the optical flow estimation. RFPM incorporates a residual structure between multiple feature pyramids into a downsampling module that corrects the blending of objects across boundaries. We demonstrate how to integrate our module with two state-of-the-art iterative refinement architectures. Results show that our RFPM visibly reduces flow errors and improves state-of-art performance in the clean pass of Sintel, and is one of the top-performing methods in KITTI. According to the particular modular structure of RFPM, we introduce a special transfer learning approach that can dramatically decrease the training time compared to a typical full optical flow training schedule on multiple datasets.