MVStylizer: An Efficient Edge-Assisted Video Photorealistic Style Transfer System for Mobile Phones

TL;DR

MVStylizer is a novel edge-assisted system that enables efficient, high-quality photorealistic video style transfer on mobile phones by combining edge computing, optical-flow-based interpolation, and federated learning.

Contribution

It introduces a new system integrating edge computing, frame interpolation, and federated learning to perform real-time video style transfer on mobile devices, overcoming computational limitations.

Findings

Achieves 75.5× speedup on 1080p videos.

Generates stylized videos with better visual quality than state-of-the-art.

Effectively leverages federated learning for continuous model improvement.

Abstract

Recent research has made great progress in realizing neural style transfer of images, which denotes transforming an image to a desired style. Many users start to use their mobile phones to record their daily life, and then edit and share the captured images and videos with other users. However, directly applying existing style transfer approaches on videos, i.e., transferring the style of a video frame by frame, requires an extremely large amount of computation resources. It is still technically unaffordable to perform style transfer of videos on mobile phones. To address this challenge, we propose MVStylizer, an efficient edge-assisted photorealistic video style transfer system for mobile phones. Instead of performing stylization frame by frame, only key frames in the original video are processed by a pre-trained deep neural network (DNN) on edge servers, while the rest of stylized intermediate frames are generated by our designed optical-flow-based frame interpolation algorithm on mobile phones. A meta-smoothing module is also proposed to simultaneously upscale a stylized frame to arbitrary resolution and remove style transfer related distortions in these upscaled frames. In addition, for the sake of continuously enhancing the performance of the DNN model on the edge server, we adopt a federated learning scheme to keep retraining each DNN model on the edge server with collected data from mobile clients and syncing with a global DNN model on the cloud server. Such a scheme effectively leverages the diversity of collected data from various mobile clients and efficiently improves the system performance. Our experiments demonstrate that MVStylizer can generate stylized videos with an even better visual quality compared to the state-of-the-art method while achieving 75.5$\times$ speedup for 1920$\times$1080 videos.