Balancing Tails when Comparing Distributions: Comprehensive Equity Index (CEI) with Application to Bias Evaluation in Operational Face Biometrics

TL;DR

The paper introduces the Comprehensive Equity Index (CEI), a new metric for detecting subtle demographic biases in face recognition systems by analyzing score distribution tails, with an automated version CEI^A for practical use.

Contribution

It presents the CEI metric that separately analyzes genuine and impostor score distributions, focusing on tails, and introduces CEI^A, an automated, objective version for bias evaluation.

Findings

CEI outperforms existing metrics in detecting subtle biases.

CEI^A simplifies bias assessment with automation.

Experiments confirm CEI's robustness across datasets and models.

Abstract

Demographic bias in high-performance face recognition (FR) systems often eludes detection by existing metrics, especially with respect to subtle disparities in the tails of the score distribution. We introduce the Comprehensive Equity Index (CEI), a novel metric designed to address this limitation. CEI uniquely analyzes genuine and impostor score distributions separately, enabling a configurable focus on tail probabilities while also considering overall distribution shapes. Our extensive experiments (evaluating state-of-the-art FR systems, intentionally biased models, and diverse datasets) confirm CEI's superior ability to detect nuanced biases where previous methods fall short. Furthermore, we present CEI^A, an automated version of the metric that enhances objectivity and simplifies practical application. CEI provides a robust and sensitive tool for operational FR fairness assessment. The proposed methods have been developed particularly for bias evaluation in face biometrics but, in general, they are applicable for comparing statistical distributions in any problem where one is interested in analyzing the distribution tails.