Sum of Group Error Differences: A Critical Examination of Bias Evaluation in Biometric Verification and a Dual-Metric Measure

TL;DR

This paper critically examines existing bias evaluation metrics in biometric verification, highlighting their limitations, and introduces a new measure called SEDG that better quantifies demographic bias, supported by experimental validation.

Contribution

It identifies limitations of current bias metrics in biometric verification and proposes a novel, general-purpose measure called SEDG for improved bias quantification.

Findings

Existing metrics have limitations in capturing bias magnitude.

SEDG effectively quantifies demographic bias in synthetic datasets.

Scenario-based recommendations improve bias evaluation practices.

Abstract

Biometric Verification (BV) systems often exhibit accuracy disparities across different demographic groups, leading to biases in BV applications. Assessing and quantifying these biases is essential for ensuring the fairness of BV systems. However, existing bias evaluation metrics in BV have limitations, such as focusing exclusively on match or non-match error rates, overlooking bias on demographic groups with performance levels falling between the best and worst performance levels, and neglecting the magnitude of the bias present. This paper presents an in-depth analysis of the limitations of current bias evaluation metrics in BV and, through experimental analysis, demonstrates their contextual suitability, merits, and limitations. Additionally, it introduces a novel general-purpose bias evaluation measure for BV, the ``Sum of Group Error Differences (SEDG)''. Our experimental results on controlled synthetic datasets demonstrate the effectiveness of demographic bias quantification when using existing metrics and our own proposed measure. We discuss the applicability of the bias evaluation metrics in a set of simulated demographic bias scenarios and provide scenario-based metric recommendations. Our code is publicly available under \url{https://github.com/alaaobeid/SEDG}.