Learning Perspective Undistortion of Portraits

TL;DR

This paper introduces a deep learning method to correct perspective distortion in portraits, improving facial recognition, reconstruction, and enabling new camera calibration techniques without relying on 3D face modeling.

Contribution

It presents the first deep learning approach to remove perspective distortion from unconstrained portraits by predicting a distortion flow map, outperforming previous methods especially on extreme distortions.

Findings

Outperforms previous state-of-the-art in distortion correction

Improves accuracy of face recognition and 3D reconstruction

Enables camera calibration from a single portrait

Abstract

Near-range portrait photographs often contain perspective distortion artifacts that bias human perception and challenge both facial recognition and reconstruction techniques. We present the first deep learning based approach to remove such artifacts from unconstrained portraits. In contrast to the previous state-of-the-art approach, our method handles even portraits with extreme perspective distortion, as we avoid the inaccurate and error-prone step of first fitting a 3D face model. Instead, we predict a distortion correction flow map that encodes a per-pixel displacement that removes distortion artifacts when applied to the input image. Our method also automatically infers missing facial features, i.e. occluded ears caused by strong perspective distortion, with coherent details. We demonstrate that our approach significantly outperforms the previous state-of-the-art both qualitatively and quantitatively, particularly for portraits with extreme perspective distortion or facial expressions. We further show that our technique benefits a number of fundamental tasks, significantly improving the accuracy of both face recognition and 3D reconstruction and enables a novel camera calibration technique from a single portrait. Moreover, we also build the first perspective portrait database with a large diversity in identities, expression and poses, which will benefit the related research in this area.