On the Reliability of Biometric Datasets: How Much Test Data Ensures Reliability?

TL;DR

This paper introduces BioQuake, a new measure for estimating uncertainty in biometric verification systems, validated across multiple datasets and modalities, highlighting the importance of reliability in performance reporting.

Contribution

The paper presents BioQuake, a novel method for quantifying uncertainty in biometric system performance, and provides guidelines for more reliable reporting of error rates.

Findings

BioQuake effectively estimates uncertainty across diverse biometric modalities.

Reported performance often significantly deviates from actual error rates.

BioQuake is available as an accessible web tool for researchers.

Abstract

Biometric authentication is increasingly popular for its convenience and accuracy. However, while recent advancements focus on reducing errors and expanding modalities, the reliability of reported performance metrics often remains overlooked. Understanding reliability is critical, as it communicates how accurately reported error rates represent a system's actual performance, considering the uncertainty in error-rate estimates from test data. Currently, there is no widely accepted standard for reporting these uncertainties and indeed biometric studies rarely provide reliability estimates, limiting comparability and interpretation. To address this gap, we introduce BioQuake--a measure to estimate uncertainty in biometric verification systems--and empirically validate it on four systems and three datasets. Based on BioQuake, we provide simple guidelines for estimating performance uncertainty and facilitating reliable reporting. Additionally, we apply BioQuake to analyze biometric recognition performance on 62 biometric datasets used in research across eight modalities: face, fingerprint, gait, iris, keystroke, eye movement, Electroencephalogram (EEG), and Electrocardiogram (ECG). Our analysis shows that reported state-of-the-art performance often deviates significantly from actual error rates, potentially leading to inaccurate conclusions. To support researchers and foster the development of more reliable biometric systems and datasets, we release BioQuake as an easy-to-use web tool for reliability calculations.