Out of a hundred trials, how many errors does your speaker verifier make?

TL;DR

This paper explains how to compute the actual error rate of speaker verification systems using Bayes decision theory, considering calibration, priors, and costs, and demonstrates this with a recent verifier.

Contribution

It introduces a method to compute the Bayes error-rate for speaker verifiers, accounting for calibration and priors, providing a more practical error measure than traditional ROC/DET curves.

Findings

Bayes error-rate bounds by min(EER, P, 1-P) for perfect calibration

Method to compute error-rate for non-perfect calibration

Analysis of error-rates for DCA-PLDA verifier

Abstract

Out of a hundred trials, how many errors does your speaker verifier make? For the user this is an important, practical question, but researchers and vendors typically sidestep it and supply instead the conditional error-rates that are given by the ROC/DET curve. We posit that the user's question is answered by the Bayes error-rate. We present a tutorial to show how to compute the error-rate that results when making Bayes decisions with calibrated likelihood ratios, supplied by the verifier, and an hypothesis prior, supplied by the user. For perfect calibration, the Bayes error-rate is upper bounded by min(EER,P,1-P), where EER is the equal-error-rate and P, 1-P are the prior probabilities of the competing hypotheses. The EER represents the accuracy of the verifier, while min(P,1-P) represents the hardness of the classification problem. We further show how the Bayes error-rate can be computed also for non-perfect calibration and how to generalize from error-rate to expected cost. We offer some criticism of decisions made by direct score thresholding. Finally, we demonstrate by analyzing error-rates of the recently published DCA-PLDA speaker verifier.