Information-Theoretic Bias Assessment Of Learned Representations Of Pretrained Face Recognition

TL;DR

This paper introduces an information-theoretic bias metric for pretrained face recognition models, addressing limitations of existing metrics and providing a benchmark for evaluating debiasing methods.

Contribution

It proposes a novel, independent bias assessment metric based on information theory, and establishes a benchmark for evaluating bias in face recognition representations.

Findings

The new metric effectively discriminates bias with small variation.

Logits-level loss is insufficient to explain bias.

The synthetic dataset mitigates sample size issues.

Abstract

As equality issues in the use of face recognition have garnered a lot of attention lately, greater efforts have been made to debiased deep learning models to improve fairness to minorities. However, there is still no clear definition nor sufficient analysis for bias assessment metrics. We propose an information-theoretic, independent bias assessment metric to identify degree of bias against protected demographic attributes from learned representations of pretrained facial recognition systems. Our metric differs from other methods that rely on classification accuracy or examine the differences between ground truth and predicted labels of protected attributes predicted using a shallow network. Also, we argue, theoretically and experimentally, that logits-level loss is not adequate to explain bias since predictors based on neural networks will always find correlations. Further, we present a synthetic dataset that mitigates the issue of insufficient samples in certain cohorts. Lastly, we establish a benchmark metric by presenting advantages in clear discrimination and small variation comparing with other metrics, and evaluate the performance of different debiased models with the proposed metric.