AB/BA analysis: A framework for estimating keyword spotting recall improvement while maintaining audio privacy

TL;DR

This paper introduces AB/BA analysis, a privacy-preserving evaluation framework for keyword spotting systems that estimates recall improvements without requiring negative samples or compromising privacy.

Contribution

The paper presents a novel AB/BA analysis framework for evaluating KWS models, including methods for estimating recall and false positive rates under privacy constraints.

Findings

AB/BA analysis accurately measures recall improvements.

The framework estimates false positive rate with low variance.

Semi-supervised extension enhances analysis efficiency and privacy.

Abstract

Evaluation of keyword spotting (KWS) systems that detect keywords in speech is a challenging task under realistic privacy constraints. The KWS is designed to only collect data when the keyword is present, limiting the availability of hard samples that may contain false negatives, and preventing direct estimation of model recall from production data. Alternatively, complementary data collected from other sources may not be fully representative of the real application. In this work, we propose an evaluation technique which we call AB/BA analysis. Our framework evaluates a candidate KWS model B against a baseline model A, using cross-dataset offline decoding for relative recall estimation, without requiring negative examples. Moreover, we propose a formulation with assumptions that allow estimation of relative false positive rate between models with low variance even when the number of false positives is small. Finally, we propose to leverage machine-generated soft labels, in a technique we call Semi-Supervised AB/BA analysis, that improves the analysis time, privacy, and cost. Experiments with both simulation and real data show that AB/BA analysis is successful at measuring recall improvement in conjunction with the trade-off in relative false positive rate.