Migrating Face Swap to Mobile Devices: A lightweight Framework and A Supervised Training Solution

TL;DR

This paper introduces MobileFSGAN, a lightweight, real-time face swap framework optimized for mobile devices, utilizing supervised training with a new dataset and multi-scale loss functions.

Contribution

The paper presents a novel lightweight GAN architecture, FSTriplets dataset for supervised training, and multi-scale gradient losses, enabling efficient face swapping on resource-constrained devices.

Findings

Achieves competitive face swap quality with significantly fewer parameters.

Runs in real-time on mobile devices.

Outperforms existing methods in efficiency while maintaining quality.

Abstract

Existing face swap methods rely heavily on large-scale networks for adequate capacity to generate visually plausible results, which inhibits its applications on resource-constraint platforms. In this work, we propose MobileFSGAN, a novel lightweight GAN for face swap that can run on mobile devices with much fewer parameters while achieving competitive performance. A lightweight encoder-decoder structure is designed especially for image synthesis tasks, which is only 10.2MB and can run on mobile devices at a real-time speed. To tackle the unstability of training such a small network, we construct the FSTriplets dataset utilizing facial attribute editing techniques. FSTriplets provides source-target-result training triplets, yielding pixel-level labels thus for the first time making the training process supervised. We also designed multi-scale gradient losses for efficient back-propagation, resulting in faster and better convergence. Experimental results show that our model reaches comparable performance towards state-of-the-art methods, while significantly reducing the number of network parameters. Codes and the dataset have been released.