Kunster -- AR Art Video Maker -- Real time video neural style transfer on mobile devices

TL;DR

This paper introduces Kunster, a real-time neural style transfer application for mobile devices that achieves stable video stylization at over 25 frames per second, making artistic video creation accessible to non-experts.

Contribution

It presents a mobile-compatible neural style transfer method for real-time video, including techniques for temporal coherence and model fine-tuning to ensure stability.

Findings

Achieves real-time stylization on iOS devices

Demonstrates stable video transfer with fine-tuning

Analyzes neural network architecture impact on mobile performance

Abstract

Neural style transfer is a well-known branch of deep learning research, with many interesting works and two major drawbacks. Most of the works in the field are hard to use by non-expert users and substantial hardware resources are required. In this work, we present a solution to both of these problems. We have applied neural style transfer to real-time video (over 25 frames per second), which is capable of running on mobile devices. We also investigate the works on achieving temporal coherence and present the idea of fine-tuning, already trained models, to achieve stable video. What is more, we also analyze the impact of the common deep neural network architecture on the performance of mobile devices with regard to number of layers and filters present. In the experiment section we present the results of our work with respect to the iOS devices and discuss the problems present in current Android devices as well as future possibilities. At the end we present the qualitative results of stylization and quantitative results of performance tested on the iPhone 11 Pro and iPhone 6s. The presented work is incorporated in Kunster - AR Art Video Maker application available in the Apple's App Store.