An Efficient Recurrent Adversarial Framework for Unsupervised Real-Time Video Enhancement

TL;DR

This paper introduces a novel recurrent adversarial framework for unsupervised real-time video enhancement that efficiently propagates spatio-temporal information, enabling high-quality enhancement at over 35 frames per second without paired training data.

Contribution

It proposes a new recurrent cell design with local and global modules for efficient spatio-temporal information integration in an unsupervised setting.

Findings

Outperforms state-of-the-art in visual quality and metrics

Achieves real-time processing at over 35 FPS for FullHD videos

Uses a single discriminator for joint domain learning

Abstract

Video enhancement is a challenging problem, more than that of stills, mainly due to high computational cost, larger data volumes and the difficulty of achieving consistency in the spatio-temporal domain. In practice, these challenges are often coupled with the lack of example pairs, which inhibits the application of supervised learning strategies. To address these challenges, we propose an efficient adversarial video enhancement framework that learns directly from unpaired video examples. In particular, our framework introduces new recurrent cells that consist of interleaved local and global modules for implicit integration of spatial and temporal information. The proposed design allows our recurrent cells to efficiently propagate spatio-temporal information across frames and reduces the need for high complexity networks. Our setting enables learning from unpaired videos in a cyclic adversarial manner, where the proposed recurrent units are employed in all architectures. Efficient training is accomplished by introducing one single discriminator that learns the joint distribution of source and target domain simultaneously. The enhancement results demonstrate clear superiority of the proposed video enhancer over the state-of-the-art methods, in all terms of visual quality, quantitative metrics, and inference speed. Notably, our video enhancer is capable of enhancing over 35 frames per second of FullHD video (1080x1920).