Recycle-GAN: Unsupervised Video Retargeting

TL;DR

Recycle-GAN is an unsupervised video retargeting method that effectively translates content across domains while maintaining style, leveraging spatiotemporal constraints and adversarial training for diverse applications.

Contribution

It introduces a novel unsupervised approach combining spatial and temporal information with adversarial losses for effective video retargeting.

Findings

Spatiotemporal constraints outperform spatial constraints in retargeting.

Successful applications include face translation, flower synthesis, and weather scene generation.

The method preserves style while accurately translating content across domains.

Abstract

We introduce a data-driven approach for unsupervised video retargeting that translates content from one domain to another while preserving the style native to a domain, i.e., if contents of John Oliver's speech were to be transferred to Stephen Colbert, then the generated content/speech should be in Stephen Colbert's style. Our approach combines both spatial and temporal information along with adversarial losses for content translation and style preservation. In this work, we first study the advantages of using spatiotemporal constraints over spatial constraints for effective retargeting. We then demonstrate the proposed approach for the problems where information in both space and time matters such as face-to-face translation, flower-to-flower, wind and cloud synthesis, sunrise and sunset.