Optical Flow Super-Resolution Based on Image Guidence Using Convolutional Neural Network

TL;DR

This paper introduces an end-to-end CNN approach for super-resolving low-resolution optical flow fields, guided by the first frame, improving resolution and accuracy over traditional interpolation methods.

Contribution

The proposed method uniquely addresses optical flow super-resolution by handling less textured data and estimation errors, outperforming existing techniques in accuracy and edge sharpness.

Findings

Achieves 15% accuracy improvement on FlyingChairs

Achieves 13% accuracy improvement on MPI Sintel

Produces sharper edges in optical flow fields

Abstract

The convolutional neural network model for optical flow estimation usually outputs a low-resolution(LR) optical flow field. To obtain the corresponding full image resolution,interpolation and variational approach are the most common options, which do not effectively improve the results. With the motivation of various convolutional neural network(CNN) structures succeeded in single image super-resolution(SISR) task, an end-to-end convolutional neural network is proposed to reconstruct the high resolution(HR) optical flow field from initial LR optical flow with the guidence of the first frame used in optical flow estimation. Our optical flow super-resolution(OFSR) problem differs from the general SISR problem in two main aspects. Firstly, the optical flow includes less texture information than image so that the SISR CNN structures can't be directly used in our OFSR problem. Secondly, the initial LR optical flow data contains estimation error, while the LR image data for SISR is generally a bicubic downsampled, blurred, and noisy version of HR ground truth. We evaluate the proposed approach on two different optical flow estimation mehods and show that it can not only obtain the full image resolution, but generate more accurate optical flow field (Accuracy improve 15% on FlyingChairs, 13% on MPI Sintel) with sharper edges than the estimation result of original method.