Video Frame Interpolation with Flow Transformer

TL;DR

This paper introduces a Flow Transformer-based framework for video frame interpolation that effectively captures motion dynamics and long-range pixel dependencies, resulting in higher quality interpolated frames especially with large motions.

Contribution

It develops a novel Flow Transformer Block incorporating optical flow-guided temporal self-attention, addressing limitations of convolutional methods in capturing details and long-range dependencies.

Findings

Outperforms state-of-the-art methods on three benchmarks

Effectively handles large motion in video frames

Maintains low complexity with local attention mechanism

Abstract

Video frame interpolation has been actively studied with the development of convolutional neural networks. However, due to the intrinsic limitations of kernel weight sharing in convolution, the interpolated frame generated by it may lose details. In contrast, the attention mechanism in Transformer can better distinguish the contribution of each pixel, and it can also capture long-range pixel dependencies, which provides great potential for video interpolation. Nevertheless, the original Transformer is commonly used for 2D images; how to develop a Transformer-based framework with consideration of temporal self-attention for video frame interpolation remains an open issue. In this paper, we propose Video Frame Interpolation Flow Transformer to incorporate motion dynamics from optical flows into the self-attention mechanism. Specifically, we design a Flow Transformer Block that calculates the temporal self-attention in a matched local area with the guidance of flow, making our framework suitable for interpolating frames with large motion while maintaining reasonably low complexity. In addition, we construct a multi-scale architecture to account for multi-scale motion, further improving the overall performance. Extensive experiments on three benchmarks demonstrate that the proposed method can generate interpolated frames with better visual quality than state-of-the-art methods.