LLM-Synth4KWS: Scalable Automatic Generation and Synthesis of Confusable Data for Custom Keyword Spotting

TL;DR

This paper presents LLM-Synth4KWS, a scalable method for generating confusable training data for custom keyword spotting using large language models and text-to-speech synthesis, improving model robustness.

Contribution

It introduces a novel data augmentation approach leveraging LLMs and TTS to generate confusable utterances, enhancing keyword spotting performance.

Findings

AUC improved by 3.7% on Speech Commands dataset

Confusable group c-AUC increased by 11.3%

Method offers scalable, zero-labor data augmentation

Abstract

Custom keyword spotting (KWS) allows detecting user-defined spoken keywords from streaming audio. This is achieved by comparing the embeddings from voice enrollments and input audio. State-of-the-art custom KWS models are typically trained contrastively using utterances whose keywords are randomly sampled from training dataset. These KWS models often struggle with confusing keywords, such as "blue" versus "glue". This paper introduces an effective way to augment the training with confusable utterances where keywords are generated and grouped from large language models (LLMs), and speech signals are synthesized with diverse speaking styles from text-to-speech (TTS) engines. To better measure user experience on confusable KWS, we define a new northstar metric using the average area under DET curve from confusable groups (c-AUC). Featuring high scalability and zero labor cost, the proposed method improves AUC by 3.7% and c-AUC by 11.3% on the Speech Commands testing set.