A Comparison of Differential Performance Metrics for the Evaluation of Automatic Speaker Verification Fairness

TL;DR

This paper compares three fairness metrics for automatic speaker verification, finding that GARBE uniquely satisfies key fairness criteria and highlighting the complex trade-offs between fairness and accuracy in system performance.

Contribution

It introduces a comparison of fairness metrics in ASV, extending prior face recognition work, and evaluates five state-of-the-art systems for fairness and accuracy.

Findings

GARBE is the only metric meeting all fairness criteria

A nuanced trade-off exists between fairness and verification accuracy

Evaluation of five ASV systems reveals complex fairness-accuracy interplay

Abstract

When decisions are made and when personal data is treated by automated processes, there is an expectation of fairness -- that members of different demographic groups receive equitable treatment. This expectation applies to biometric systems such as automatic speaker verification (ASV). We present a comparison of three candidate fairness metrics and extend previous work performed for face recognition, by examining differential performance across a range of different ASV operating points. Results show that the Gini Aggregation Rate for Biometric Equitability (GARBE) is the only one which meets three functional fairness measure criteria. Furthermore, a comprehensive evaluation of the fairness and verification performance of five state-of-the-art ASV systems is also presented. Our findings reveal a nuanced trade-off between fairness and verification accuracy underscoring the complex interplay between system design, demographic inclusiveness, and verification reliability.