FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation

TL;DR

FLAVR introduces a 3D convolution-based video frame interpolation method that models complex motions and occlusions without optical flow, achieving faster inference and superior results across multiple benchmarks.

Contribution

The paper presents FLAVR, a novel end-to-end trainable architecture using 3D convolutions for efficient, flow-agnostic video frame interpolation that outperforms flow-based methods.

Findings

3x faster inference than current top methods

Superior qualitative and quantitative results on benchmarks

Effective as a self-supervised pretext task for related tasks

Abstract

A majority of methods for video frame interpolation compute bidirectional optical flow between adjacent frames of a video, followed by a suitable warping algorithm to generate the output frames. However, approaches relying on optical flow often fail to model occlusions and complex non-linear motions directly from the video and introduce additional bottlenecks unsuitable for widespread deployment. We address these limitations with FLAVR, a flexible and efficient architecture that uses 3D space-time convolutions to enable end-to-end learning and inference for video frame interpolation. Our method efficiently learns to reason about non-linear motions, complex occlusions and temporal abstractions, resulting in improved performance on video interpolation, while requiring no additional inputs in the form of optical flow or depth maps. Due to its simplicity, FLAVR can deliver 3x faster inference speed compared to the current most accurate method on multi-frame interpolation without losing interpolation accuracy. In addition, we evaluate FLAVR on a wide range of challenging settings and consistently demonstrate superior qualitative and quantitative results compared with prior methods on various popular benchmarks including Vimeo-90K, UCF101, DAVIS, Adobe, and GoPro. Finally, we demonstrate that FLAVR for video frame interpolation can serve as a useful self-supervised pretext task for action recognition, optical flow estimation, and motion magnification.