Attributes Shape the Embedding Space of Face Recognition Models

TL;DR

This paper investigates how facial and image attributes influence the geometric structure of face recognition embedding spaces, revealing varying degrees of invariance and interpretability of models.

Contribution

It introduces a geometric, physics-inspired metric to analyze attribute dependence in face recognition models, enhancing understanding of their invariance properties.

Findings

Models show different invariance levels to facial attributes.

The proposed metric effectively measures attribute dependence.

Insights into model strengths and weaknesses are gained.

Abstract

Face Recognition (FR) tasks have made significant progress with the advent of Deep Neural Networks, particularly through margin-based triplet losses that embed facial images into high-dimensional feature spaces. During training, these contrastive losses focus exclusively on identity information as labels. However, we observe a multiscale geometric structure emerging in the embedding space, influenced by interpretable facial (e.g., hair color) and image attributes (e.g., contrast). We propose a geometric approach to describe the dependence or invariance of FR models to these attributes and introduce a physics-inspired alignment metric. We evaluate the proposed metric on controlled, simplified models and widely used FR models fine-tuned with synthetic data for targeted attribute augmentation. Our findings reveal that the models exhibit varying degrees of invariance across different attributes, providing insight into their strengths and weaknesses and enabling deeper interpretability. Code available here: https://github.com/mantonios107/attrs-fr-embs}{https://github.com/mantonios107/attrs-fr-embs