Frame Interpolation for Dynamic Scenes with Implicit Flow Encoding

TL;DR

This paper introduces a novel frame interpolation method for dynamic scenes that effectively handles illumination changes by encoding optical flow with a hypernetwork, resulting in superior intermediate frame quality.

Contribution

It presents a new approach that encodes bidirectional optical flows into a coordinate-based network using a hypernetwork, improving interpolation in scenes with lighting variations.

Findings

Outperforms state-of-the-art frame interpolation methods

Effectively handles brightness and illumination changes

Produces higher quality intermediate frames

Abstract

In this paper, we propose an algorithm to interpolate between a pair of images of a dynamic scene. While in the past years significant progress in frame interpolation has been made, current approaches are not able to handle images with brightness and illumination changes, which are common even when the images are captured shortly apart. We propose to address this problem by taking advantage of the existing optical flow methods that are highly robust to the variations in the illumination. Specifically, using the bidirectional flows estimated using an existing pre-trained flow network, we predict the flows from an intermediate frame to the two input images. To do this, we propose to encode the bidirectional flows into a coordinate-based network, powered by a hypernetwork, to obtain a continuous representation of the flow across time. Once we obtain the estimated flows, we use them within an existing blending network to obtain the final intermediate frame. Through extensive experiments, we demonstrate that our approach is able to produce significantly better results than state-of-the-art frame interpolation algorithms.