Learning to Transfer Visual Effects from Videos to Images

TL;DR

This paper presents a method for transferring spatio-temporal visual effects from videos to images, ensuring effects are applied without altering content or style, enabling realistic animations like melting or blooming.

Contribution

We introduce a novel approach that captures visual effects using specific loss functions and constrains pixel movement to transfer effects accurately from videos to images.

Findings

Effective transfer of visual effects demonstrated in qualitative results

Optical flow and texture motion are key to capturing effects

Pixel movement constraint prevents unwanted style transfer

Abstract

We study the problem of animating images by transferring spatio-temporal visual effects (such as melting) from a collection of videos. We tackle two primary challenges in visual effect transfer: 1) how to capture the effect we wish to distill; and 2) how to ensure that only the effect, rather than content or artistic style, is transferred from the source videos to the input image. To address the first challenge, we evaluate five loss functions; the most promising one encourages the generated animations to have similar optical flow and texture motions as the source videos. To address the second challenge, we only allow our model to move existing image pixels from the previous frame, rather than predicting unconstrained pixel values. This forces any visual effects to occur using the input image's pixels, preventing unwanted artistic style or content from the source video from appearing in the output. We evaluate our method in objective and subjective settings, and show interesting qualitative results which demonstrate objects undergoing atypical transformations, such as making a face melt or a deer bloom.