The BOSARIS Toolkit: Theory, Algorithms and Code for Surviving the New DCF

TL;DR

The BOSARIS Toolkit provides advanced algorithms and tools for calibration, evaluation, and fusion of speaker recognition systems under the challenging new DCF criterion, enabling effective handling of large datasets.

Contribution

It introduces novel algorithms and tools for calibration, evaluation, and fusion tailored to the new DCF, including visualization, efficient computation, and large-scale data handling.

Findings

Normalized Bayes Error-Rate Plot for calibration analysis

Efficient algorithms for large score files and DCF computation

Enhanced logistic regression optimizer for system fusion

Abstract

The change of two orders of magnitude in the 'new DCF' of NIST's SRE'10, relative to the 'old DCF' evaluation criterion, posed a difficult challenge for participants and evaluator alike. Initially, participants were at a loss as to how to calibrate their systems, while the evaluator underestimated the required number of evaluation trials. After the fact, it is now obvious that both calibration and evaluation require very large sets of trials. This poses the challenges of (i) how to decide what number of trials is enough, and (ii) how to process such large data sets with reasonable memory and CPU requirements. After SRE'10, at the BOSARIS Workshop, we built solutions to these problems into the freely available BOSARIS Toolkit. This paper explains the principles and algorithms behind this toolkit. The main contributions of the toolkit are: 1. The Normalized Bayes Error-Rate Plot, which analyses likelihood- ratio calibration over a wide range of DCF operating points. These plots also help in judging the adequacy of the sizes of calibration and evaluation databases. 2. Efficient algorithms to compute DCF and minDCF for large score files, over the range of operating points required by these plots. 3. A new score file format, which facilitates working with very large trial lists. 4. A faster logistic regression optimizer for fusion and calibration. 5. A principled way to define EER (equal error rate), which is of practical interest when the absolute error count is small.