Improving Optical Flow on a Pyramid Level

TL;DR

This paper enhances optical flow estimation by improving pyramid level processing, cost volume construction, gradient flow, and knowledge distillation, leading to state-of-the-art results on multiple datasets.

Contribution

It introduces a sampling-based cost volume strategy, a level-specific loss max-pooling, gradient blocking techniques, and a distillation method for sequential training, advancing optical flow accuracy.

Findings

Achieves new state-of-the-art on Sintel and KITTI datasets.

Demonstrates improved convergence and flow detail preservation.

Shows portability of methods to stereo depth estimation.

Abstract

In this work we review the coarse-to-fine spatial feature pyramid concept, which is used in state-of-the-art optical flow estimation networks to make exploration of the pixel flow search space computationally tractable and efficient. Within an individual pyramid level, we improve the cost volume construction process by departing from a warping- to a sampling-based strategy, which avoids ghosting and hence enables us to better preserve fine flow details. We further amplify the positive effects through a level-specific, loss max-pooling strategy that adaptively shifts the focus of the learning process on under-performing predictions. Our second contribution revises the gradient flow across pyramid levels. The typical operations performed at each pyramid level can lead to noisy, or even contradicting gradients across levels. We show and discuss how properly blocking some of these gradient components leads to improved convergence and ultimately better performance. Finally, we introduce a distillation concept to counteract the issue of catastrophic forgetting and thus preserving knowledge over models sequentially trained on multiple datasets. Our findings are conceptually simple and easy to implement, yet result in compelling improvements on relevant error measures that we demonstrate via exhaustive ablations on datasets like Flying Chairs2, Flying Things, Sintel and KITTI. We establish new state-of-the-art results on the challenging Sintel and KITTI 2012 test datasets, and even show the portability of our findings to different optical flow and depth from stereo approaches.