Joint Face Image Restoration and Frontalization for Recognition

TL;DR

This paper introduces MDFR, a unified model that jointly restores and frontalizes low-quality face images with arbitrary poses, improving face recognition performance in challenging real-world conditions.

Contribution

MDFR is a novel encoder-decoder network that simultaneously restores and frontalizes faces, incorporating pose residual learning and a 3D-based normalization for better identity preservation.

Findings

Outperforms state-of-the-art methods in face frontalization.

Effectively handles arbitrary poses and low-quality factors.

Enhances face recognition accuracy in real-world scenarios.

Abstract

In real-world scenarios, many factors may harm face recognition performance, e.g., large pose, bad illumination,low resolution, blur and noise. To address these challenges, previous efforts usually first restore the low-quality faces to high-quality ones and then perform face recognition. However, most of these methods are stage-wise, which is sub-optimal and deviates from the reality. In this paper, we address all these challenges jointly for unconstrained face recognition. We propose an Multi-Degradation Face Restoration (MDFR) model to restore frontalized high-quality faces from the given low-quality ones under arbitrary facial poses, with three distinct novelties. First, MDFR is a well-designed encoder-decoder architecture which extracts feature representation from an input face image with arbitrary low-quality factors and restores it to a high-quality counterpart. Second, MDFR introduces a pose residual learning strategy along with a 3D-based Pose Normalization Module (PNM), which can perceive the pose gap between the input initial pose and its real-frontal pose to guide the face frontalization. Finally, MDFR can generate frontalized high-quality face images by a single unified network, showing a strong capability of preserving face identity. Qualitative and quantitative experiments on both controlled and in-the-wild benchmarks demonstrate the superiority of MDFR over state-of-the-art methods on both face frontalization and face restoration.