Learning Energy Based Inpainting for Optical Flow

TL;DR

This paper introduces an interpretable, energy-based inpainting algorithm for optical flow that is lightweight, modular, and competitive with state-of-the-art methods, utilizing a novel optimization layer for efficient training.

Contribution

It presents a new inpainting-based optical flow method with an optimization layer enabling backpropagation through many iterations, improving interpretability and efficiency.

Findings

Competitive accuracy with state-of-the-art methods

Lightweight and modular CNN architecture

Efficient backpropagation through 10K iterations

Abstract

Modern optical flow methods are often composed of a cascade of many independent steps or formulated as a black box neural network that is hard to interpret and analyze. In this work we seek for a plain, interpretable, but learnable solution. We propose a novel inpainting based algorithm that approaches the problem in three steps: feature selection and matching, selection of supporting points and energy based inpainting. To facilitate the inference we propose an optimization layer that allows to backpropagate through 10K iterations of a first-order method without any numerical or memory problems. Compared to recent state-of-the-art networks, our modular CNN is very lightweight and competitive with other, more involved, inpainting based methods.