Facial Soft Biometrics for Recognition in the Wild: Recent Works, Annotation, and COTS Evaluation

TL;DR

This paper investigates the use of soft biometrics like gender, ethnicity, and age to improve face recognition in unconstrained environments, demonstrating significant performance gains when combined with deep learning-based systems.

Contribution

It evaluates the discrimination power of soft biometrics alone and in fusion with state-of-the-art face recognition systems, providing publicly available annotations and COTS outputs.

Findings

Soft biometrics improve verification performance by up to 40% with manual estimation.

Fusion of soft biometrics with face recognition yields up to 15% performance improvement.

Public datasets and results facilitate reproducibility and further research.

Abstract

The role of soft biometrics to enhance person recognition systems in unconstrained scenarios has not been extensively studied. Here, we explore the utility of the following modalities: gender, ethnicity, age, glasses, beard, and moustache. We consider two assumptions: 1) manual estimation of soft biometrics and 2) automatic estimation from two commercial off-the-shelf systems (COTS). All experiments are reported using the labeled faces in the wild (LFW) database. First, we study the discrimination capabilities of soft biometrics standalone. Then, experiments are carried out fusing soft biometrics with two state-of-the-art face recognition systems based on deep learning. We observe that soft biometrics is a valuable complement to the face modality in unconstrained scenarios, with relative improvements up to 40%/15% in the verification performance when using manual/automatic soft biometrics estimation. Results are reproducible as we make public our manual annotations and COTS outputs of soft biometrics over LFW, as well as the face recognition scores.