OFER: Occluded Face Expression Reconstruction

TL;DR

OFER introduces a diffusion-based method for reconstructing diverse, plausible 3D face models from a single occluded image, effectively handling ambiguity and generating expressive faces.

Contribution

The paper presents a novel diffusion model approach for multi-hypothesis 3D face reconstruction under occlusion, including a ranking mechanism and a new dataset CO-545.

Findings

Outperforms existing occlusion-based methods in accuracy

Generates diverse and expressive 3D face reconstructions

Introduces CO-545 dataset for occluded face evaluation

Abstract

Reconstructing 3D face models from a single image is an inherently ill-posed problem, which becomes even more challenging in the presence of occlusions. In addition to fewer available observations, occlusions introduce an extra source of ambiguity where multiple reconstructions can be equally valid. Despite the ubiquity of the problem, very few methods address its multi-hypothesis nature. In this paper we introduce OFER, a novel approach for single-image 3D face reconstruction that can generate plausible, diverse, and expressive 3D faces, even under strong occlusions. Specifically, we train two diffusion models to generate the shape and expression coefficients of a face parametric model, conditioned on the input image. This approach captures the multi-modal nature of the problem, generating a distribution of solutions as output. However, to maintain consistency across diverse expressions, the challenge is to select the best matching shape. To achieve this, we propose a novel ranking mechanism that sorts the outputs of the shape diffusion network based on predicted shape accuracy scores. We evaluate our method using standard benchmarks and introduce CO-545, a new protocol and dataset designed to assess the accuracy of expressive faces under occlusion. Our results show improved performance over occlusion-based methods, while also enabling the generation of diverse expressions for a given image.