Pay Attention to Hard Trials

TL;DR

This paper highlights the importance of focusing on hard trials in speaker recognition evaluation, proposing an SVM-based method to identify such trials and create more challenging benchmarks for fairer system comparison.

Contribution

It introduces a novel SVM-based approach to identify hard trials and constructs new evaluation sets that improve the assessment of speaker recognition systems.

Findings

New evaluation sets with hard trials improve system benchmarking.

Re-evaluation of recent technologies shows different performance insights.

Code and datasets will be publicly available.

Abstract

Performance of speaker recognition systems is evaluated on test trials. Although as crucial as rulers for tailors, trials have not been carefully treated so far, and most existing benchmarks compose trials by naive cross-pairing. In this paper, we argue that the cross-pairing approach produces overwhelming easy trials, which in turn leads to potential bias in system and technique comparison. To solve the problem, we advocate more attention to hard trials. We present an SVM-based approach to identifying hard trials and use it to construct new evaluation sets for VoxCeleb1 and SITW. With the new sets, we can re-evaluate the contribution of some recent technologies. The code and the identified hard trials will be published online at http://project.cslt.org.