FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors

TL;DR

This paper introduces FSRNet, an end-to-end deep learning framework that leverages facial priors like landmarks and parsing maps to improve face super-resolution, especially for very low-resolution images, and enhances realism with adversarial training.

Contribution

The paper proposes a novel face super-resolution network that integrates facial priors and adversarial training, advancing the state-of-the-art in very low-resolution face image enhancement.

Findings

FSRNet outperforms existing methods quantitatively and qualitatively.

Incorporating facial priors improves super-resolution quality.

The proposed FSRGAN generates more realistic face images.

Abstract

Face Super-Resolution (SR) is a domain-specific super-resolution problem. The specific facial prior knowledge could be leveraged for better super-resolving face images. We present a novel deep end-to-end trainable Face Super-Resolution Network (FSRNet), which makes full use of the geometry prior, i.e., facial landmark heatmaps and parsing maps, to super-resolve very low-resolution (LR) face images without well-aligned requirement. Specifically, we first construct a coarse SR network to recover a coarse high-resolution (HR) image. Then, the coarse HR image is sent to two branches: a fine SR encoder and a prior information estimation network, which extracts the image features, and estimates landmark heatmaps/parsing maps respectively. Both image features and prior information are sent to a fine SR decoder to recover the HR image. To further generate realistic faces, we propose the Face Super-Resolution Generative Adversarial Network (FSRGAN) to incorporate the adversarial loss into FSRNet. Moreover, we introduce two related tasks, face alignment and parsing, as the new evaluation metrics for face SR, which address the inconsistency of classic metrics w.r.t. visual perception. Extensive benchmark experiments show that FSRNet and FSRGAN significantly outperforms state of the arts for very LR face SR, both quantitatively and qualitatively. Code will be made available upon publication.