PatchDSU: Uncertainty Modeling for Out of Distribution Generalization in Keyword Spotting

TL;DR

This paper introduces PatchDSU, a novel method that improves out-of-distribution generalization in keyword spotting by splitting spectrogram inputs into patches and augmenting them independently, addressing domain shifts effectively.

Contribution

PatchDSU extends DSU by partitioning input spectrograms into patches for independent augmentation, enhancing robustness to distribution shifts in speech recognition tasks.

Findings

PatchDSU outperforms DSU and other methods in most scenarios.

PatchDSU shows more consistent improvements across various datasets.

Both PatchDSU and DSU improve out-of-domain generalization.

Abstract

Deep learning models excel at many tasks but rely on the assumption that training and test data follow the same distribution. This assumption often does not hold in real-world speech systems, where distribution shifts are common due to varying environments, recording conditions, and speaker diversity. The method of Domain Shifts with Uncertainty (DSU) augments the input of each neural network layer based on the input feature statistics. It addresses the problem of out-of-domain generalization by assuming feature statistics follow a multivariate Gaussian distribution and substitutes the input with sampled features from this distribution. While effective for computer vision, applying DSU to speech presents challenges due to the nature of the data. Unlike static visual data, speech is a temporal signal commonly represented by a spectrogram - the change of frequency over time. This representation cannot be treated as a simple image, and the resulting sparsity can lead to skewed feature statistics when applied to the entire input. To tackle out-of-distribution issues in keyword spotting, we propose PatchDSU, which extends DSU by splitting the input into patches and independently augmenting each patch. We evaluated PatchDSU and DSU alongside other methods on the Google Speech Commands, Librispeech, and TED-LIUM. Additionally, we evaluated performance under white Gaussian and MUSAN music noise conditions. We also explored out-of-domain generalization by analyzing model performance on datasets they were not trained on. Overall, in most cases, both PatchDSU and DSU outperform other methods. Notably, PatchDSU demonstrates more consistent improvements across the evaluated scenarios compared to other approaches.