C-P Map: A Novel Evaluation Toolkit for Speaker Verification

TL;DR

This paper introduces C-P map, a new evaluation toolkit for automatic speaker verification systems that provides detailed performance analysis across various trial configurations, addressing biases in traditional evaluation methods.

Contribution

The paper presents a theoretical framework and the C-P map visualization tool to improve the evaluation and comparison of speaker verification systems.

Findings

C-P map effectively visualizes system performance across trial configurations.

It reveals biases in traditional evaluation methods.

Experiments demonstrate its usefulness for performance analysis.

Abstract

Evaluation trials are used to probe performance of automatic speaker verification (ASV) systems. In spite of the clear importance and impact, evaluation trials have not been seriously treated in research and engineering practice. This paper firstly presents a theoretical analysis on evaluation trials and highlights potential bias with the most popular cross-pairing approach used in trials design. To interpret and settle this problem, we define the concept of trial config and C-P map derived from it. The C-P map measures the performance of an ASV system on various trial configs in a 2-dimensional map. On the map, each location represents a particular trial config and its corresponding color represents the system performance. Experiments conducted on representative ASV systems show that the proposed C-P map offers a powerful evaluation toolkit for ASV performance analysis and comparison. The source code for C-P map has been release at https://gitlab.com/csltstu/sunine.