Video Enhancement with Task-Oriented Flow

TL;DR

This paper introduces task-oriented flow (TOFlow), a self-supervised neural network approach that learns motion representations tailored for specific video enhancement tasks, outperforming traditional optical flow methods.

Contribution

The paper presents a novel self-supervised learning framework for task-specific motion estimation, jointly trained with video processing modules, and introduces a new large-scale dataset Vimeo-90K.

Findings

TOFlow outperforms traditional optical flow in benchmarks

Effective for frame interpolation, denoising, and super-resolution

Provides a task-specific motion representation for video enhancement

Abstract

Many video enhancement algorithms rely on optical flow to register frames in a video sequence. Precise flow estimation is however intractable; and optical flow itself is often a sub-optimal representation for particular video processing tasks. In this paper, we propose task-oriented flow (TOFlow), a motion representation learned in a self-supervised, task-specific manner. We design a neural network with a trainable motion estimation component and a video processing component, and train them jointly to learn the task-oriented flow. For evaluation, we build Vimeo-90K, a large-scale, high-quality video dataset for low-level video processing. TOFlow outperforms traditional optical flow on standard benchmarks as well as our Vimeo-90K dataset in three video processing tasks: frame interpolation, video denoising/deblocking, and video super-resolution.